JP2013153256A - Terminal device, photographing system, terminal position detection method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a terminal device capable of accurately detect an arrangement state of the terminal device and other terminal devices without making the device extensive.SOLUTION: A terminal device includes: a photographing unit 8 that photographs a subject; a communication unit 4 that receives image data obtained by photographing the subject from at least one other terminal device; and a CPU 1 that detects an arrangement state of the terminal device and the other terminal device with respect to the subject on the basis of the image data of the subject photographed by the photographing unit 8 and the image data of the subject received by the communication unit 4.

Description

  The present invention relates to a technology of a terminal device represented by a mobile phone terminal such as a mobile phone or a smartphone, a tablet terminal, a notebook personal computer, and the like, and in particular, a positional relationship between each terminal device when a plurality of terminal devices are arranged side by side. It is related with the technology to detect.

  As a method for using a terminal device including a display unit such as a mobile phone or a smartphone, there is a method in which a plurality of terminal devices are arranged side by side and used as a multi-screen.

  For example, two mobile phones are arranged in the horizontal direction, and two pages of data are displayed on the left and right display units. According to this form, it can be used like a spread book.

  Two mobile phones are arranged in the vertical direction, a soft keyboard (touch keyboard) is displayed on the lower display unit, and input information from the soft keyboard is displayed on the upper display unit. According to this form, it can be used like a personal computer.

  Furthermore, four mobile phones are arranged in a matrix of 2 rows × 2 columns, and one image is divided into four partial images in the upper left, upper right, lower left, and lower right, and each partial image is assigned to the corresponding position. Display on the display. According to this form, an image can be observed on a large screen.

  There exists a portable terminal of patent document 1 as a terminal device in which said multi-screen conversion is possible.

  The portable terminal described in Patent Literature 1 includes a key input unit, a receiving unit that receives image information, a display unit that displays image information, a communication unit that communicates with another portable terminal, The determination means for determining whether or not the mobile terminal is in a predetermined arrangement, and when the own terminal and another mobile terminal are in the predetermined arrangement, the display of the own terminal corresponding to the predetermined arrangement is displayed. Control means for determining a screen portion to be displayed and causing the display means to display an image corresponding to the determined screen portion of the received image information.

  The determination means includes first and second magnetic sensors arranged on the upper left side and lower right side of the screen, and first and second magnets arranged on the upper right side and lower left side of the screen, respectively.

  When the first magnetic sensor detects the second magnet of another mobile terminal, the determination unit determines that the predetermined arrangement is in place, and when the screen is viewed, It is determined that it is located on the right side. On the other hand, when the second magnetic sensor detects the first magnet of another mobile terminal, the determination means determines that the arrangement is a predetermined arrangement, and when the mobile terminal sees the screen, It is determined that it is located on the left side of the terminal.

  There is also an image display system described in Patent Document 2.

  An image display system described in Patent Document 2 displays a plurality of display devices arranged in a matrix, each displaying identification information for identifying the device itself, and each display device receives identification information. Based on the imaging device that captures the entire multi-display in the displayed state, the identification information of each display device in the image captured by the imaging device, and the positional relationship thereof, the relative relationship of each display device in the matrix arrangement And a control device that detects a positional relationship and notifies each display device of the detection result.

  Each display device specifies the position of its own device in the matrix arrangement based on the relative positional relationship notified from the control device, and displays image information corresponding to the specified position.

JP 2011-114352 A JP 2009-109671 A

  In the portable terminal described in Patent Document 1, the following problems occur.

  When two mobile terminals are used side by side as a multi-screen, for example, when viewed from the user, the form of observing the multi-screen with the screen of each mobile terminal adjacent to the left and right, and the screen of each mobile terminal up and down There is a form in which multiple screens are observed in an adjacent state. The portable terminal described in Patent Document 1 is premised on observation in one of these forms. When three or more portable terminals are combined, or each portable terminal is not adjacent. It is difficult to correctly detect the positional relationship between the own terminal and other portable terminals with a free arrangement, such as when they are far away.

  As described above, the portable terminal described in Patent Document 1 has a problem in that it is impossible to accurately detect the arrangement state of the own terminal and another portable terminal with respect to the user.

  In the device described in Patent Literature 2, an imaging device that captures the entire multi-display and a relative positional relationship between the display devices are detected separately from the display device that constitutes the multi-display (this corresponds to the terminal device). Therefore, there is a problem that the apparatus becomes large.

  An object of the present invention is to provide a terminal device, an imaging system, a terminal position detection method, and a program capable of accurately detecting the arrangement state of the own terminal device and another terminal device without the device becoming large-scale There is to do.

In order to achieve the above object, according to one aspect of the present invention,
Photographing means for photographing the subject;
Receiving means for receiving image data obtained by photographing the subject from at least one other terminal device;
Detecting means for detecting an arrangement state of the terminal device and the other terminal device with respect to the subject based on the image data of the subject photographed by the photographing means and the image data of the subject received by the receiving means; A terminal device is provided.

According to another aspect of the invention,
At least one first processing device;
A second processing device capable of communicating with the first processing device;
The first processing device includes:
A first photographing means for photographing a subject;
Transmitting means for transmitting the first image data photographed by the first photographing means,
The second processing device includes:
A second photographing means for photographing the subject;
Receiving means for receiving the first image data;
Detecting means for detecting an arrangement state of the terminal device with respect to the subject and the first terminal device based on the first image data and the second image data photographed by the second photographing device; An imaging system is provided.

According to yet another aspect of the invention,
Take a picture of the subject with the shooting means,
Receiving image data of the subject from at least one other terminal device;
Provided is a terminal position detection method for detecting an arrangement state of the terminal device and the other terminal device with respect to the subject based on the subject image data photographed by the photographing means and the received subject image data. The

According to another aspect of the invention,
A process of shooting a subject with a shooting means;
Processing for receiving image data obtained by photographing the subject from at least one other terminal device;
Based on the image data of the subject photographed by the photographing means and the received image data of the subject, the computer executes processing for detecting the arrangement state of the terminal device and the other terminal device with respect to the subject. A program is provided.

  ADVANTAGE OF THE INVENTION According to this invention, the arrangement | positioning state of the own terminal device and another terminal device with respect to a to-be-photographed object can be detected correctly.

  Since there is no need to provide an external control device or photographing device, the device does not become large.

It is a block diagram which shows the structure of the terminal device which is one Embodiment of this invention. One procedure of the apparatus position detection process performed in the terminal device shown in FIG. 1 is shown. It is a schematic diagram which shows an example of the utilization form of the multiscreen using the terminal device shown in FIG. It is a schematic diagram for demonstrating the position detection process in the utilization form shown in FIG. It is a figure which shows an example of the detection result of the direction of a face. It is a schematic diagram for demonstrating the difference of the x-axis direction of the two terminal devices, and the difference of the y-axis direction. It is a schematic diagram for demonstrating the difference of the z-axis direction of two terminal devices. It is a schematic diagram for demonstrating the difference of the z-axis direction of two terminal devices. It is a schematic diagram for demonstrating the difference of the rotation direction around each of the x-axis of the two terminal devices, y-axis, and z-axis.

  Next, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a terminal device according to an embodiment of the present invention.

  Referring to FIG. 1, the terminal device is a terminal device represented by a mobile phone, a smartphone, or the like, and includes a CPU (Central Processing Unit) 1, a recording memory 2, a work memory 3, a communication unit 4, an operation unit 5, A display control unit 6, a display unit 7, and a photographing unit 8 are included.

  The CPU 1, main memory 2, work memory 3, communication unit 4, operation unit 5, display control unit 6, and photographing unit 8 are each connected to a bus 9. The CPU 1 controls the operation of each unit via the bus 9.

  The main memory 2 is a main storage device such as a semiconductor memory, and holds a control program for controlling the terminal device and data necessary for the control. The control program includes a program that executes various processes such as image processing, apparatus position detection processing, and multi-screen processing.

  The work memory 3 is a RAM (Random Access Memory) or the like, and temporarily holds a part of the program and necessary data when the control program is executed.

  Note that the memory unit of the terminal device is not limited to the main memory 2 and the work memory 3 described above, and may be configured such that the same memory serves as both the main memory 2 and the work memory 3. For example, the main memory 2 and the work memory 3 may be configured by a single flash memory. In this case, the flash memory includes an area (or block) corresponding to the main memory 2 and an area (or block) corresponding to the work memory 3.

  The communication unit 4 includes an inter-device communication unit that transmits and receives data to and from other terminal devices. The inter-device communication means is a wireless communication means represented by, for example, Bluetooth (registered trademark). In addition, it is also possible to use communication means using infrared communication as inter-device communication means.

  Moreover, the communication part 4 may have another communication means for connecting to the network represented by the internet other than the communication apparatus between apparatuses. Another communication means may be a communication means using a wireless local area network (LAN), for example. Necessary programs and data may be downloaded via the other communication means.

  The operation unit 5 is input means such as a plurality of operation keys and a touch panel, and supplies an operation signal corresponding to an input operation by the user to the CPU 1.

  The display unit 7 is a display device such as an LCD (Liquid Crystal Display). The display control unit 6 controls the display unit 7 in accordance with a control signal from the CPU 1. The display unit 7 and the display control unit 6 constitute display means.

  The imaging unit 8 includes an image sensor typified by a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and receives an image signal obtained via the image sensor in accordance with a control signal from the CPU 1. Supply to CPU1.

  The CPU 1 executes a control program stored in the main memory 2, receives a signal corresponding to the input operation from the operation unit 5, and controls each part of the terminal device and necessary processing according to the received signal. Execute.

  For example, the CPU 1 includes processing units (image processing unit, device position detection processing unit, and multi-screen processing unit) that execute the following image processing (shooting / saving / playback), device position detection processing, and multi-screen processing, respectively. .

(1) Image processing (shooting / saving / playback):
The operation unit 5 includes a shooting button and the like. When the user presses the shooting button on the operation unit 5, a signal indicating that the shooting button has been pressed is supplied from the operation unit 5 to the CPU 1. The CPU 1 controls the photographing unit 8 in accordance with a signal from the operation unit 5 to execute processing such as photographing a subject and capturing image data.

  In photographing a subject, for example, the CPU 1 temporarily stores the image data from the photographing unit 8 in the work memory 3, resizes the stored image data to a display size, and causes the display unit 7 to display the image data. Also good.

  When the user performs an input operation for saving a captured image on the operation unit 5, a signal corresponding to the input operation is supplied from the operation unit 5 to the CPU 1. The CPU 1 temporarily stores the image data of the photographed image in the work memory 3 according to the signal from the operation unit 5, compresses the stored image data, and converts it into an image file such as a JPEG file. Store in memory 2. Information (such as a shooting date) for identifying the file is given to the image file.

  When the user performs an input operation for reproducing a JPEG file on the operation unit 5, a signal corresponding to the input operation is supplied from the operation unit 5 to the CPU 1. The CPU 1 causes the display unit 7 to display a list of JPEG files held in the main memory 2 in response to a signal from the operation unit 5.

  When the user performs an input operation for selecting an arbitrary JPEG file from the list of JPEG files on the operation unit 5, a signal corresponding to the input operation is supplied from the operation unit 5 to the CPU 1. In response to a signal from the operation unit 5, the CPU 1 resizes the image data of the selected JPEG file to the display size and causes the display unit 7 to display it.

(2) Device position detection processing:
The apparatus position detection process includes a first detection process when the own terminal apparatus operates as a main apparatus, and a second detection process when the own terminal apparatus operates as a sub apparatus. One terminal device among the plurality of terminal devices is a main device, and the remaining terminal devices are sub devices.

  When the user performs an input operation for detecting the apparatus positional relationship using the operation unit 5, a signal corresponding to the input operation is supplied from the operation unit 5 to the CPU 1. In response to a signal from the operation unit 5, the CPU 1 executes device position detection processing with another terminal device.

  FIG. 2 shows a procedure of device position detection processing performed between two terminal devices. In FIG. 2, steps M1 to M6 indicate a processing procedure in the main apparatus, and steps S1 to S4 indicate a processing procedure in the sub apparatus. In the following description, both the main device and the sub device have the configuration shown in FIG.

  The apparatus position detection process will be specifically described below with reference to FIGS. 1 and 2.

  First, the CPU 1 establishes communication with another terminal device arranged adjacent to its own terminal device via the communication unit 4, and between the own terminal device and the other terminal device, Communication negotiation for determining the relationship between the device and the sub device is performed (step M1). In the communication negotiation, for example, a terminal device on which an input operation for detecting a device positional relationship is performed is a main device, and another terminal device is a sub device.

  In step M1, when an input operation for detecting the device positional relationship is performed, the CPU 1 sets the terminal device as the main device and the other terminal devices as sub devices.

  Next, the CPU 1 causes the photographing unit 8 to perform camera photographing, and transmits a photographing timing signal to the sub apparatus via the communication unit 4 (step M2). In camera shooting, the CPU 1 performs the same processing as the shooting described in “(1) Image processing (shooting / saving / playback)” above, and temporarily stores image data of an arbitrary subject in the work memory 3.

  Next, the CPU 1 receives the image data obtained by photographing the subject from the sub device (step M3), and receives each of the received image data of the sub device and the image data of the own terminal device stored in the work memory 3. Then, the direction of a predetermined part of the subject is detected (step M4).

  In the detection of the direction of the predetermined part of the subject, for example, a plurality of reference parts on the predetermined part are set, and the position on the predetermined part in the reference direction is set in advance for each reference part. For example, when the predetermined part is a face, the plurality of reference parts are given by two or more parts among parts such as eyes, mouths, noses, ears, and eyebrows. The CPU 1 acquires the position of each reference portion on a predetermined part for each image data, and detects the difference with respect to the reference direction based on the difference between the acquired position and a preset position. And CPU1 determines the direction of a predetermined | prescribed site | part based on the detected difference.

  Next, the CPU 1 obtains a difference between the direction of the predetermined part of the subject obtained from the image data of the terminal device and the direction of the predetermined part of the subject obtained from the image data of the sub apparatus, and based on the difference. Then, the arrangement state (specifically, the relative positional relationship) of the own terminal device (main device) and the sub device with respect to the subject is detected, and information on the detection result is transmitted to the sub device (step M5).

  Finally, the CPU 1 performs settings such as screen display based on the arrangement state (relative positional relationship) between the terminal device (main device) and the other terminal device (sub device) detected in step M5 ( Step M6).

  On the other hand, the CPU 1 of the sub apparatus executes the processes of steps S1 to S4 in conjunction with the processes of steps M1 to M6 described above.

  In step S1, the CPU 1 performs communication negotiation. In step S <b> 2, the CPU 1 receives the shooting timing from the main device via the communication unit 4, and causes the shooting unit 8 to shoot the subject at the received shooting timing. In this camera shooting, the CPU 1 performs the same processing as the shooting described in “(1) Image processing (shooting / saving / playback)”. Here, the camera shooting subject in step S2 is the same as the camera shooting subject in step M2.

  In step S3, the CPU 1 transmits the image data of the subject obtained in step S2 to the main apparatus via the communication unit 4.

  Finally, in step S4, the CPU 1 receives the detection result of the arrangement state (relative position) from the main device via the communication unit 4, and sets the screen display and the like based on the received detection result.

  In the apparatus position detection process described above, the subject may be a person or may be something other than a person. When the subject is a person, any part of the face, hand, upper body, and the like can be used as the predetermined part in step M4. Further, instead of the predetermined part, the direction of the whole body of the subject may be detected.

  Hereinafter, apparatus position detection processing in the case where two mobile phones are arranged adjacent to each other, the subject is a person, and the predetermined part is a face will be specifically described.

  As shown in FIG. 3, two mobile phones 10 and 11 are arranged adjacent to each other. Both the mobile phones 10 and 11 have the configuration shown in FIG.

  In the following, reference is made to FIGS.

  The mobile phone 10 has a display surface 10a, and a photographing lens 10b is provided on the surface on which the display surface 10a is provided. The photographing lens 10b is disposed to face the image sensor of the photographing unit 8 shown in FIG. 2, and the image sensor photographs a subject through the photographing lens 10b.

  The mobile phone 11 has a display surface 11a, and a photographing lens 11b is provided on the surface on which the display surface 11a is provided. The photographing lens 11b is disposed to face the image sensor of the photographing unit 8 shown in FIG. 2, and the image sensor photographs a subject through the photographing lens 11b.

  In the example shown in FIG. 3, the mobile phones 10 and 11 are arranged so that the display surfaces 10a and 11a are located on the same side. When the cellular phones 10 and 11 are viewed from the display surfaces 10a and 11a, the cellular phone 10 is located on the left side and the cellular phone 11 is located on the right side.

  When the user performs an input operation for detecting the positional relationship of the apparatus with the operation unit 5 of the mobile phone 10, the mobile phone 10 performs the processing of steps M1 to M6 shown in FIG. 11, the processing of steps S1 to S4 shown in FIG.

  In the mobile phone 10, the CPU 1 causes the photographing unit 8 to photograph the face of the subject in the camera photographing in step M <b> 2. Similarly, in the mobile phone 11, the CPU 1 causes the photographing unit 8 to photograph the face of the subject in the camera photographing in step S <b> 2.

  When the distance between the mobile phones 10 and 11 and the subject is short to some extent, as shown in FIG. 4, the mobile phone 10 obtains face image data 10 c obtained by photographing the subject from the left front. Face image data 11c photographed from the right front is obtained.

  In the mobile phone 10, in step M4, the CPU 1 detects the face direction of the face image data 10c and also detects the face direction of the face image data 11c.

  FIG. 5 shows an example of the detection results of the face orientations of the face image data 10c and 11c. In the example shown in FIG. 5, for the face image data 10c, the face is inclined 15 degrees to the right and 5 degrees downward, and for the face image data 11c, the face is 10 degrees to the left and downward. Tilt by 5 degrees. The CPU 1 determines that the cellular phone 10 is located on the left side and the cellular phone 10 is located on the right side when the cellular phones 10 and 11 are viewed from the subject side based on the difference between the tilt angles of the faces.

  It should be noted that the face orientations of the two image data do not necessarily have to be different from each other in the left-right direction such that the orientation of one face of the face image data 10c, 11c is the right direction and the other face is the left direction. As long as the face angles are different from each other, the face orientations of the face image data 10c and 11c may both be in the left direction (or right direction). In any case, one can determine that one is the left side and the other is the right side with respect to the subject due to the difference in the face angles of the face image data 10c and 11c.

  In the vertical direction, since the face angles of the face image data 10c and 11c are both 5 degrees, the CPU 1 determines that there is no relative displacement in the vertical direction of the mobile phones 10 and 11 with respect to the subject.

When the face image data obtained by the camera photographing in step M2 or step S2 includes a plurality of faces, the CPU 1 performs any one of the following processes (a) to (c) or those processes: The target face may be determined by executing two or more processes.
(A) The face closest to the center of the image is determined as the face to be compared.
(B) The largest face on the image is determined as the face to be compared.
(C) Of the plurality of faces, a face that matches the pre-registered face feature data is determined as a face to be compared (individual face recognition function).

(3) Multi-screen processing:
In the terminal device shown in FIG. 1, the CPU 1 on the main device side performs multi-screen display after the setting of screen display or the like in step M6 or step S4 in the above “(2) Device position detection process”. Is displayed on the display unit 7.

  When the user performs an input operation for performing multi-screen display on the operation unit 5, the operation unit 5 supplies an operation signal for multi-screen display to the CPU 1. The CPU 1 executes multi-screen display processing according to an operation signal from the operation unit 5.

  In the multi-screen display processing, the CPU 1 of the main device displays a predetermined image corresponding to the arrangement state (positional relationship) on the display unit 7 of the own device and the display unit 7 of the sub device. For example, the CPU 1 of the main device divides an image to be displayed into a plurality of partial images corresponding to the main device and the sub device according to the arrangement state, and displays the display unit 7 of the main device and the display unit 7 of the sub device. The corresponding partial images are displayed.

  Each of the main device and the sub device displays partial image data corresponding to the arrangement state (positional relationship), and a multi-screen is provided based on these display screens.

  In multi-screen display, at least one of the plurality of terminal devices may display an image different from the display image of the other terminal device. For example, when performing multi-screen display using two terminal devices, one terminal device displays map data, and data related to the map data displayed by the other terminal device (detailed map data, etc.) is displayed. May be.

  The image to be displayed is arbitrarily selected by the user from image data (still image, moving image) stored in the main memory 2 or image data (still image, moving image) acquired from an external device (server, etc.) via the communication unit 4 The selected image data may be used. Further, the image to be displayed may be image data used for an application such as a game or navigation.

  According to the terminal device of the present embodiment, it is possible to accurately detect the arrangement state (relative positional relationship) of each terminal device with respect to the subject. Therefore, for example, in multi-screen display, each terminal device can be appropriately displayed when viewed from the subject.

  Also, according to the terminal device of the present embodiment, there is no need to provide a control device or a photographing device outside, so that the device does not become large when performing multi-screen display.

  Furthermore, many recent terminal devices such as mobile phones and smartphones are equipped with a camera. The terminal device according to the present embodiment can be applied to a device including such a camera. Thereby, it is not necessary to provide a new camera, and cost reduction can be realized.

  In the terminal device according to the present embodiment, for example, two terminal devices are arranged in the horizontal direction, and two pages of data are displayed on the left and right display units (usage form like a spread book). Are arranged in a vertical direction, a soft keyboard (touch type keyboard) is displayed on the lower display unit, and input information from the soft keyboard is displayed on the upper display unit. Arranging in a matrix of rows x 2 columns, dividing one image into four partial images in the upper left, upper right, lower left, lower right, and displaying each partial image on the display unit at the corresponding position, etc. It can take various forms of multi-screen display.

  The terminal device of the present embodiment described above is an example of the present invention, and the configuration thereof can be changed as appropriate.

  For example, in step M4 in FIG. 2, the CPU 1 may detect the difference in the image sensor direction (position or inclination difference) between the main device and the sub device from both images.

  For example, in the three-dimensional space defined by the xyz axis, the difference in the direction of the image sensor is the x axis direction, the y axis direction, the z axis direction, the rotation direction around the x axis, the rotation direction around the y axis, and the rotation direction around the z axis Is given by the difference of the six directional components.

  FIG. 6 is a schematic diagram for explaining the difference in the x-axis direction and the difference in the y-axis direction.

  Referring to FIG. 6, the image sensor 10d constitutes a photographing unit (the photographing unit 8 shown in FIG. 1) of the main device, and the image sensor 11d is a photographing unit (the photographing unit 8 shown in FIG. 1) of the sub device. ).

  The light receiving surfaces of the image sensors 10d and 11d are parallel to a plane (photographing plane) facing the face. When viewed from the direction perpendicular to the photographing plane, the difference between the upper end of the image sensor 10d and the upper end of the image sensor 11d is a y-direction difference, and the right end of the image sensor 10d and the left end of the image sensor 11d. Is the difference in the x direction.

  7 and 8 are schematic diagrams for explaining the difference in the z-axis direction.

  7 and 8, image sensors 10 d and 11 d constitute the imaging units (imaging unit 8 shown in FIG. 1) of the main device and the sub device, respectively.

  The light receiving surfaces of the image sensors 10d and 11d are parallel to a plane facing the face, and the direction perpendicular to the plane is the z-axis direction. The difference in the z-axis direction between the light receiving surface of the image sensor 10d and the light receiving surface of the image sensor 11d is a z-direction difference.

  FIG. 9 is a schematic diagram for explaining the difference in the rotation direction around each of the x-axis, the y-axis, and the z-axis.

  In FIG. 9, image sensors 10d and 11d constitute the imaging units (imaging unit 8 shown in FIG. 1) of the main device and the sub device, respectively. The vertical direction is the y-axis direction, the horizontal direction is the x-axis direction, and the direction perpendicular to the plane is the z-axis direction.

  A state in which the light receiving surfaces of the image sensors 10d and 11d are parallel to a plane facing the face, and the upper and lower ends and the left and right ends of each light receiving surface are horizontal and vertical, respectively, is a reference state. The angle difference from the reference state in the rotation direction around the y axis is the difference in the rotation direction around the y axis. The angle difference from the reference state in the rotation direction around the x axis is the difference in the rotation direction around the x axis. The angle difference from the reference state in the rotation direction around the z axis is the difference in the rotation direction around the z axis.

  The arrangement state of the main device and the sub device when viewed from the subject may be detected based on at least one of the six direction components.

  By using the difference between the x-axis direction and the y-axis direction, the relative positional relationship between the main device and the sub device with respect to the subject can be specified. In this case, in multi-screen display, an appropriate image can be displayed on each display unit of the main device and the sub device based on the positional relationship.

  By using the difference in the z-axis direction, the front-rear positional relationship between the main device and the sub device when viewed from the subject can be specified. In this case, in the multi-screen display, the CPU of the main device controls the magnification of the display image of the sub device with respect to the display image of the main device according to the front-rear positional relationship. For example, when the sub device is located behind the main device, the display image of the sub device is enlarged or the display image of the main device is reduced. On the other hand, when the sub device is positioned on the near side of the main device, the display image of the main device is enlarged or the display image of the sub device is reduced. This makes it possible to align the display images of the main device and the sub device when viewed from the subject.

  By using the difference in the rotation direction around the x axis, it is possible to detect the front / rear tilt of the main device and the sub device when viewed from the subject. When the main device or the sub device tilts forward or backward, the image is enlarged in the vertical direction according to the tilt angle, or trapezoidal correction is applied to the image. Here, the trapezoid correction is a technique well known in projectors and the like, which is a correction that makes the image display area trapezoidal so that the image display area when viewed from the subject has a rectangular shape. Thereby, it is possible to observe the display images of the main device and the sub device in an appropriate shape when viewed from the subject.

  By using the difference in the rotation direction around the y-axis, it is possible to detect the left and right inclinations of the main device and the sub device when viewed from the subject. When the main device or the sub device is tilted to the left or right, the image is enlarged in the horizontal direction according to the tilt angle, or trapezoidal correction is applied to the image. Thereby, it is possible to observe the display images of the main device and the sub device in an appropriate shape when viewed from the subject.

  By using the difference in the rotation direction around the z-axis, it is possible to detect the clockwise or counterclockwise rotation angle of the main device and the sub device when viewed from the subject. When the main device or the sub device rotates clockwise or counterclockwise, the image is rotated clockwise or counterclockwise according to the rotation angle. Accordingly, it is possible to correct the rotation direction of the display image of each of the main device and the sub device when viewed from the subject.

  Further, in the main apparatus, the processes of steps M1 to M6 in FIG. 2 may be repeatedly executed at regular time intervals. Thereby, for example, when a plurality of people are observing a multi-screen, and display setting is performed according to the arrangement state of the main device and the sub device with respect to the person (target) close to the center position of the captured image When the target person stands, the arrangement state of the main device and the sub device with respect to the person close to the center position among the remaining people is automatically detected, and the display setting according to the arrangement state is performed. Done.

  In the apparatus position detection process, the arrangement state of the terminal apparatus with respect to the subject may be detected by a method using the following unified condition.

  The shooting conditions of each terminal device are unified in advance. Specifically, the pitch angle of the terminal device with respect to the ground in the shooting direction (tilt in the vertical direction, that is, the rotation angle around the x-axis shown in FIG. 9) is substantially 0, and the rotation angle around the axis in the shooting direction (ie The rotation angle around the z-axis shown in FIG. That is, it unifies on the condition that it shoots in a direction horizontal to the ground without tilting the terminal device.

  According to this unified condition, only the yaw angle (tilt in the left-right direction, that is, the rotation angle around the y-axis shown in FIG. 9) with respect to the ground in the shooting direction should be considered as the angle component in the shooting direction.

  For example, when the subject includes a square surface (hereinafter, referred to as a square part) and the terminal device captures a square part under the above-described unified condition, the arrangement state of the terminal device with respect to the square part is calculated as follows. be able to. In the following description, the terminal device and the square part are arranged at a predetermined distance (predetermined distance), and the square part is set with the rotation angle, the pitch angle, and the yaw angle about the photographing direction axis set to 0, respectively. It is assumed that the size of the square part on the captured image is given in advance as reference information.

  When the terminal device captures a square part under the uniform condition, the length of the vertical side of the square part on the captured image is inversely proportional to the distance between the terminal device and the square part. Since the length of the vertical side of the square part on the photographed image when the terminal device and the square part are arranged at a predetermined distance (imaging distance) is given in advance as reference information, using the reference information, The photographing distance when the distance between the terminal device and the square part is unknown can be calculated. Specifically, when the distance between the terminal device and the square part is unknown, the distance between the terminal device and the square part based on the length of the vertical side of the square part on the captured image and the reference information Can be calculated.

  Furthermore, the length of the horizontal side of the square part on the photographed image is inversely proportional to the distance between the terminal device and the square part, and the angle between the photographing direction and the perpendicular from the surface of the square part is θ. It is a value multiplied by cos θ. Here, the angle θ corresponds to the yaw angle.

  Since the distance between the terminal device and the square part can be calculated from the length of the vertical side of the square part on the photographed image, the photographing direction and the square can be calculated from the length of the horizontal side of the square part on the photographed image. The angle θ formed with the surface of the part can be calculated.

For example, assuming that the length of the side of the square part on the photographed image when the distance between the terminal device and the square part is L1, the pitch angle θx with respect to the ground in the photographing direction is 0 degree. In this case, the distance L between the terminal device and the square part is given by the following formula 1. In Expression 1, “the length of the vertical side of the square part on the photographed image” is the length of the vertical side of the square part on the photographed image photographed at the distance L.
(Formula 1)
L = (a1 ÷ (length of the vertical side of the square part on the photographed image) ÷ cos0) × L1
= (A1 ÷ (length of vertical side of square part on photographed image)) × L1
If the yaw angle with respect to the ground in the photographing direction is θy, the distance L between the terminal device and the square part is given by the following equation 2. In Equation 2, “the length of the side of the square part on the photographed image” is the length of the side of the square part on the photographed image photographed at the distance L.
(Formula 2)
L = (a1 ÷ (length of the side of the square part on the photographed image) ÷ cos θy) × L1
Based on the above formulas 1 and 2, the angle θy, that is, the arrangement state of the terminal device with respect to the square part (the direction of the square part on the captured image) can be calculated.

  Here, the length on the image of the vertical side of the photographed square part is converted into the actual length because the square part photographed when the distance between the terminal device and the square part is known in advance. It can be converted from the length of the vertical or horizontal side on the photographed image.

  In the terminal device that is the main device, the CPU uses the above-described method using the unified condition, and based on the image data photographed by the own terminal device and the image data photographed by the sub device, each terminal device for the square portion Is calculated (the direction of the square part on the captured image), and based on the calculation result, the arrangement state (relative positional relationship) of the terminal device and the sub-terminal with respect to the square part is determined. Note that the values of “a1” and “L1” in the above formulas 1 and 2 are given in advance as reference information, and these values are the image data photographed by the terminal device and the image data photographed by the sub device. Can be used in common.

  Further, it is possible to further add the pitch angle θx as an unknown parameter under the above unified condition. In this case, when the terminal device captures a square portion that is a subject, the shape of the subject on the captured image is deformed into a parallel square when both the pitch angle θx and the yaw angle θy are not zero. The lengths of the adjacent first and second sides of the parallelogram subject (which correspond to the vertical and horizontal sides of the original square, respectively), and the lengths of the first and second sides. The formed angle changes according to the pitch angle θx and the yaw angle θy.

  When the terminal device and the square part are arranged at a predetermined distance and the square part is photographed with the rotation angle, the pitch angle, and the yaw angle about the photographing direction axis set to 0, the square part on the photographed image is displayed. The dimensions (lengths of the first and second sides) are assumed to be given in advance as reference information.

  The distance L between the terminal device and the square part, the pitch angle θx, and the yaw angle θy are all unknown parameters. From the reference information, the lengths of the first and second sides on the photographed image, and the angles formed by the first and second sides, the distance L between the terminal device and the square portion, the pitch angle θx, and the yaw The angle θy can be obtained. And the arrangement | positioning state (for example, relative positional relationship) of the terminal device with respect to a square site | part is acquirable based on the calculated pitch angle (theta) x and yaw angle (theta) y.

  Further, in the method using the above-described unified condition, a part such as a face or a designated color part may be used instead of the square part. In either case, the first to third reference parts or the first to fourth reference parts are set.

  When the first to third reference portions are set, the first straight portion connecting between the first and second reference portions corresponds to the first side of the square portion. A second straight line portion connecting the first line portion and the third reference portion and perpendicular to the first straight line portion corresponds to the second side of the square portion. For example, when using a face, one eye can be set as a first reference part, the other eye as a second reference part, and the mouth as a third reference part.

  When the first to fourth reference portions are set, the first straight portion connecting the first and second reference portions corresponds to the first side of the square portion. A second straight line portion connecting the third and fourth reference portions and perpendicular to the first straight line portion corresponds to the first side of the square portion.

  The terminal device according to the present embodiment can be applied to various terminal devices such as a mobile phone terminal such as a mobile phone and a smartphone, a tablet terminal, and a notebook personal computer, and an electronic device (for example, a game machine) that operates independently. ).

  Further, the terminal device of the present embodiment may be configured using a computer system in which a computer operates according to a program. The program can cause a computer to execute at least the position detection process described above. The program may be provided using a computer-readable recording medium or may be provided via a communication network (for example, the Internet).

The present invention may take forms such as the following supplementary notes 1 to 22, but is not limited to these forms.
(Appendix 1)
Photographing means for photographing the subject;
Receiving means for receiving image data obtained by photographing the subject from at least one other terminal device;
Detecting means for detecting an arrangement state of the terminal device and the other terminal device with respect to the subject based on the image data of the subject photographed by the photographing means and the image data of the subject received by the receiving means; A terminal device.
(Appendix 2)
The detecting means detects the direction of the subject in the image from each of the image data photographed by the photographing means and the image data received by the receiving means, and based on the difference between the detection directions, the arrangement state The terminal device according to appendix 1, wherein a positional relationship between the terminal device and the other terminal device with respect to the subject is detected.
(Appendix 3)
The terminal device according to attachment 2, wherein the direction of the subject is a direction of a predetermined part of the subject.
(Appendix 4)
The predetermined portion includes a surface including first and second straight portions orthogonal to each other, or a surface capable of setting the first and second straight portions,
The detection means includes a first rotation angle around a first axis perpendicular to the imaging surface of the photographing means, and a second rotation angle around a second axis perpendicular to the first axis. And the predetermined portion from the direction perpendicular to the surface is determined on the condition that the third rotation angle around the third axis perpendicular to the first and second axes is zero. Holding reference information including information on the lengths of the first and second straight line portions and the predetermined distance in a photographed image obtained by photographing at a distance of
Both the image data of the subject photographed by the photographing means and the image data of the subject received by the receiving means are image data photographed under a condition in which both the first and second rotation angles are 0,
The detection means sets the predetermined distance and the length of the first straight line portion included in the reference information as L1 and a1, respectively, and sets the shooting distance as L for each of the image data, and the third The rotation angle is θ, and the lengths of the first and second linear portions on the captured image based on the image data are a2 and b1, respectively.
L = (a1 ÷ b1) × L1
L = (a1 ÷ a2 ÷ cos θ) × L1
The terminal device according to appendix 3, wherein the direction of the subject in the captured image is detected based on the formula:
(Appendix 5)
The predetermined portion includes a surface including first and second straight portions orthogonal to each other, or a surface capable of setting the first and second straight portions,
The detection means includes a first rotation angle around a first axis perpendicular to the imaging surface of the photographing means, and a second rotation angle around a second axis perpendicular to the first axis. And the predetermined portion from the direction perpendicular to the surface is determined on the condition that the third rotation angle around the third axis perpendicular to the first and second axes is zero. Holding reference information including information on the lengths of the first and second straight line portions and the predetermined distance in a photographed image obtained by photographing at a distance of
Both the image data of the subject photographed by the photographing means and the image data of the subject received by the receiving means are image data photographed under the condition that the first rotation angle is 0,
For each of the image data, the detection means includes lengths of the first and second straight line portions on the photographed image based on the image data, angles formed by the first and second straight line portions, and the reference information. The terminal device according to attachment 3, wherein the direction of the subject in the captured image is detected based on
(Appendix 6)
The terminal device according to any one of appendices 3 to 5, wherein the predetermined part is a face.
(Appendix 7)
Based on the difference between the detection directions, the detection means has, as the arrangement state, a positional relationship of a light receiving surface of a photographing means provided in the other terminal device with respect to a light receiving surface of the photographing means when viewed from the subject. The terminal device according to appendix 1, which detects a difference in angle between the light receiving surface and the reference surface.
(Appendix 8)
The terminal device according to any one of appendices 1 to 7, further comprising a transmission unit configured to transmit the information on the arrangement state detected by the detection unit to the other terminal device.
(Appendix 9)
Display means for displaying an image;
The display unit according to any one of appendices 1 to 8, further comprising: a display control unit configured to display a predetermined image corresponding to the arrangement state on the display unit and the display unit provided in the other terminal device. Terminal device.
(Appendix 10)
The display control unit divides an image to be displayed into a plurality of partial images corresponding to the terminal device and the other terminal device according to the arrangement state, and displays the image on the display unit and the other terminal device. The terminal device according to appendix 9, wherein partial images respectively corresponding to provided display means are displayed.
(Appendix 11)
At least one first processing device;
A second processing device capable of communicating with the first processing device;
The first processing apparatus includes:
A first photographing means for photographing a subject;
Transmitting means for transmitting the first image data photographed by the first photographing means,
The second processing apparatus includes:
A second photographing means for photographing the subject;
Receiving means for receiving the first image data;
Detecting means for detecting an arrangement state of the terminal device with respect to the subject and the first terminal device based on the first image data and the second image data taken by the second photographing device; Including shooting system.
(Appendix 12)
The detection means detects the direction of the subject in the image from each of the first image data and the second image data, and sets the arrangement state based on the difference between the detection directions as the arrangement state. The imaging system according to appendix 11, which detects a positional relationship between the first and second processing devices.
(Appendix 13)
The imaging system according to attachment 12, wherein the direction of the subject is a direction of a predetermined part of the subject.
(Appendix 14)
The predetermined portion includes a surface including first and second straight portions orthogonal to each other, or a surface capable of setting the first and second straight portions,
The detection means includes a first rotation angle around an axis of a first axis perpendicular to the imaging surface of the second imaging means, and a second around an axis of a second axis perpendicular to the first axis. And the third rotation angle around the axis of the third axis perpendicular to the first and second axes are both set to 0 from the direction perpendicular to the surface. Reference information including information on the lengths of the first and second straight line portions and the predetermined distance in a photographed image obtained by photographing a part at a predetermined distance,
The second image data is image data taken under a condition in which both the first and second rotation angles are 0,
The first image data is image data taken under a condition in which both the first and second rotation angles in the first photographing means are 0,
The detection means sets the predetermined distance and the length of the first straight line portion included in the reference information as L1 and a1, respectively, and sets the shooting distance as L for each of the first and second image data. And the third rotation angle is θ, and the lengths of the first and second linear portions on the captured image based on the image data are a2 and b1, respectively.
L = (a1 ÷ b1) × L1
L = (a1 ÷ a2 ÷ cos θ) × L1
14. The photographing system according to appendix 13, wherein the direction of the subject in the photographed image is detected based on the formula:
(Appendix 15)
The predetermined portion includes a surface including first and second straight portions orthogonal to each other, or a surface capable of setting the first and second straight portions,
The detection means includes a first rotation angle around an axis of a first axis perpendicular to the imaging surface of the second imaging means, and a second around an axis of a second axis perpendicular to the first axis. And the third rotation angle around the axis of the third axis perpendicular to the first and second axes are both set to 0 from the direction perpendicular to the surface. Reference information including information on the lengths of the first and second straight line portions and the predetermined distance in a photographed image obtained by photographing a part at a predetermined distance,
The second image data is image data taken under the condition that the first rotation angle is 0,
The first image data is image data captured under a condition that the first rotation angle in the first imaging unit is 0,
For each of the first and second image data, the detection means includes the lengths of the first and second straight portions on the captured image based on the image data and the first and second straight portions. 14. The photographing system according to appendix 13, wherein the direction of the subject in the photographed image is detected based on an angle formed and the reference information.
(Appendix 16)
The imaging system according to any one of appendices 13 to 15, wherein the predetermined part is a face.
(Appendix 17)
Based on the difference between the detection directions, the detection means has a positional relationship and an angle of the light receiving surface of the first imaging unit with respect to the light receiving surface of the second imaging unit when viewed from the subject as the arrangement state. The imaging system according to appendix 11, wherein a difference between the two is detected.
(Appendix 18)
The imaging system according to any one of appendices 11 to 17, wherein the second processing device further includes a transmission unit configured to transmit information on the arrangement state detected by the detection unit to the first processing device. .
(Appendix 19)
The first processing apparatus further includes first display means for displaying an image,
The second processing apparatus includes:
Second display means for displaying an image;
The imaging system according to any one of appendices 11 to 18, further comprising display control means for displaying a predetermined image corresponding to the arrangement state on the first display means and the second display means. .
(Appendix 20)
The display control unit divides an image to be displayed into a plurality of partial images corresponding to the first and second processing devices according to the arrangement state, and the first display unit and the second display unit are divided. 20. The photographing system according to appendix 19, wherein partial images respectively corresponding to the display means are displayed.
(Appendix 21)
Take a picture of the subject
Receiving image data of the subject from at least one other terminal device;
A terminal position detection method for detecting an arrangement state of the terminal device and the other terminal device with respect to the subject based on the image data of the subject photographed by the photographing means and the received image data of the subject.
(Appendix 22)
A process of shooting a subject with a shooting means;
Processing for receiving image data obtained by photographing the subject from at least one other terminal device;
Based on the image data of the subject photographed by the photographing means and the received image data of the subject, the computer executes processing for detecting the arrangement state of the terminal device and the other terminal device with respect to the subject. program.

  Note that in the terminal device of Supplementary Note 1, the photographing unit, the receiving unit, and the detecting unit correspond to the photographing unit 8, the communication unit 4, and the CPU 1 shown in FIG. The CPU 1 includes a processing unit corresponding to the detection unit.

  In the terminal device of appendix 8, the transmission means corresponds to the communication unit 14 shown in FIG.

  In the terminal device of appendix 9, the display unit corresponds to the display unit including the display control unit 6 and the display unit 7 shown in FIG. 1, and the display control unit corresponds to the CPU 1 shown in FIG. The CPU 1 includes a processing unit corresponding to the display control unit.

  In the imaging system according to attachment 11, the first processing device and the second processing device respectively correspond to the sub device and the main device described with reference to FIG.

1: CPU
2: main memory 3: work memory 4: communication unit 5: operation unit 6: display control unit 7: display unit 8: photographing unit 9: bus

Claims (10)

Photographing means for photographing the subject;
Receiving means for receiving image data obtained by photographing the subject from at least one other terminal device;
Detecting means for detecting an arrangement state of the terminal device and the other terminal device with respect to the subject based on the image data of the subject photographed by the photographing means and the image data of the subject received by the receiving means; A terminal device.   The detecting means detects the direction of the subject in the image from each of the image data photographed by the photographing means and the image data received by the receiving means, and based on the difference between the detection directions, the arrangement state The terminal device according to claim 1, wherein a positional relationship between the terminal device and the other terminal device with respect to the subject is detected.   The terminal device according to claim 2, wherein the direction of the subject is a direction of a predetermined part of the subject. The predetermined portion includes a surface including first and second straight portions orthogonal to each other, or a surface capable of setting the first and second straight portions,
The detection means includes a first rotation angle around a first axis perpendicular to the imaging surface of the photographing means, and a second rotation angle around a second axis perpendicular to the first axis. And the predetermined portion from the direction perpendicular to the surface is determined on the condition that the third rotation angle around the third axis perpendicular to the first and second axes is zero. Holding reference information including information on the lengths of the first and second straight line portions and the predetermined distance in a photographed image obtained by photographing at a distance of
Both the image data of the subject photographed by the photographing means and the image data of the subject received by the receiving means are image data photographed under a condition in which both the first and second rotation angles are 0,
The detection means sets the predetermined distance and the length of the first straight line portion included in the reference information as L1 and a1, respectively, and sets the shooting distance as L for each of the image data, and the third The rotation angle is θ, and the lengths of the first and second linear portions on the captured image based on the image data are a2 and b1, respectively.
L = (a1 ÷ b1) × L1
L = (a1 ÷ a2 ÷ cos θ) × L1
The terminal device according to claim 3, wherein the direction of the subject in the photographed image is detected based on the formula:   The terminal device according to claim 3 or 4, wherein the predetermined part is a face.   6. The terminal device according to claim 1, further comprising a transmission unit configured to transmit information on the arrangement state detected by the detection unit to the other terminal device. Display means for displaying an image;
7. The display control unit according to claim 1, further comprising: a display control unit configured to display a predetermined image corresponding to the arrangement state on the display unit and a display unit provided in the other terminal device. Terminal equipment. At least one first processing device;
A second processing device capable of communicating with the first processing device;
The first processing device includes:
A first photographing means for photographing a subject;
Transmitting means for transmitting the first image data photographed by the first photographing means,
The second processing device includes:
A second photographing means for photographing the subject;
Receiving means for receiving the first image data;
Detecting means for detecting an arrangement state of the terminal device with respect to the subject and the first terminal device based on the first image data and the second image data photographed by the second photographing device; Including shooting system. Take a picture of the subject with the shooting means,
Receiving image data of the subject from at least one other terminal device;
A terminal position detection method for detecting an arrangement state of the own terminal device and the other terminal device with respect to the subject based on the subject image data photographed by the photographing means and the received subject image data. A process of shooting a subject with a shooting means;
Processing for receiving image data obtained by photographing the subject from at least one other terminal device;
Based on the image data of the subject photographed by the photographing means and the received image data of the subject, the computer executes processing for detecting the arrangement state of the terminal device and the other terminal device with respect to the subject. program.

Images