JP2017199980A - Image display system, information processing apparatus, imaging apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display system capable of evaluating an operation of a moving object easily.SOLUTION: An image display system having an information processing apparatus, multiple wide angle imaging apparatuses capable of imaging the circumference in the horizontal direction, respectively, and a measuring apparatus for measuring the operation of a moving object, further includes acquisition means for acquiring multiple pieces of image data generated by capturing the object with the multiple wide angle imaging apparatuses, respectively, and measurement data generated by measuring the operation of the object with the measuring apparatus, mapping means for mapping the multiple pieces of image data, respectively, to the measurement data, determination means for determining the image data of display object, out of the multiple pieces of image data mapped to the measurement data, respectively, and display means for displaying the determined image data and the measurement data in a display device.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image display system, an information processing apparatus, an imaging apparatus, and a program.

  2. Description of the Related Art A motion capture system that analyzes a person's walking motion is conventionally known. In the motion capture system, for example, a person wearing a predetermined marker at a joint position or the like of a body is imaged with a camera, and the movement of the person can be analyzed by tracking the movement of the marker with a tracker.

  Here, the motion capture system requires a large number of markers in order to capture human movements from various angles, and there is a tendency that it takes time and cost to prepare for imaging of walking motion and analysis after imaging. In addition, it is difficult to analyze walking motions except in a special environment such as a laboratory.

  For these problems, a technique is known that can analyze walking motion with a relatively simple configuration using captured images of a pedestrian with a plurality of marks attached to a predetermined part of the body (for example, Patent Documents). 1).

  However, the above-described conventional technique has a problem that the walking motion that can be imaged is limited. This is because the position and orientation of the camera are fixed while the person gradually moves by walking. For example, when the camera is fixed, it is possible to capture a walking motion other than the walking motion of a pedestrian walking in a straight line (for example, a walking motion of a pedestrian walking while turning around the camera). Have difficulty.

  Embodiments of the present invention have been made in view of the above points, and an object of the present invention is to provide an image display system that can easily evaluate the movement of a moving object.

  In order to achieve the above object, an embodiment of the present invention is an image including an information processing device, a plurality of wide-angle imaging devices capable of imaging the periphery of the horizontal direction, and a measuring device that measures the movement of a moving object. In the display system, a plurality of image data generated by the plurality of wide-angle imaging devices respectively capturing the target, and measurement data generated by the measurement device measuring the operation of the target are acquired. An acquisition unit; an association unit that associates each of the plurality of image data with the measurement data; and a determination that determines image data to be displayed among the plurality of image data respectively associated with the measurement data Means, and display means for displaying the determined image data and measurement data on a display device.

  According to the embodiment of the present invention, it is possible to provide an image display system that can easily evaluate the movement of a moving object.

It is a figure explaining an example of schematic operation | movement of an image display system. It is a figure which shows an example of the system configuration | structure of an image display system. It is a figure which shows an example of the hardware constitutions of a server and a display apparatus. It is a figure which shows an example of the hardware constitutions of an acceleration measuring device. It is a figure which shows an example of the hardware constitutions of an omnidirectional image imaging device. It is a figure which shows an example of the functional block of a server, a display apparatus, an acceleration measuring apparatus, and an omnidirectional image imaging device. It is a figure explaining an example of a pedestrian's walk. It is a sequence diagram explaining an example of the whole operation | movement of an image display system. It is a sequence diagram explaining the modification (the 1) of the whole operation | movement of an image display system. It is a sequence diagram explaining the modification (the 2) of the whole operation | movement of an image display system. It is a sequence diagram explaining the modification (the 3) of the whole operation | movement of an image display system. FIG. 10 is a sequence diagram illustrating a modification (No. 4) of the overall operation of the image display system. It is a sequence diagram explaining the modification (the 5) of the whole operation | movement of an image display system. It is a flowchart explaining an example of a display image creation process. It is a figure which illustrates typically an example of specification of a rectangular field in which a pedestrian is reflected. It is a flowchart explaining the modification of a display image creation process. It is a figure explaining the other example of a pedestrian's walk. It is a figure which shows an example of the display image data and measurement data which were displayed on the display apparatus. It is a flowchart which shows an example of the process which displays the range which the pedestrian is reflected in an image data column. It is a figure which illustrates typically an example of the position of the foot of a pedestrian in the top view of an omnidirectional image. It is a flowchart which shows the other example of the process which displays the range where the pedestrian is reflected in the image data column. It is a figure which illustrates an example of recognition of a face typically. It is a figure which shows the other example of the system configuration | structure of an image display system. It is a sequence diagram explaining an example of the whole operation | movement of an image display system in case an acceleration measuring apparatus displays measurement data and display image data.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Outline of image display system>
First, a schematic operation of the image display system 100 according to the present embodiment will be described. FIG. 1 is a diagram illustrating an example of a schematic operation of the image display system 100 according to the present embodiment.

  In the image display system 100 according to the present embodiment, two omnidirectional image capturing devices 60A and 60B are installed, and a pedestrian 200 walking around the line L is treated as an omnidirectional image capturing device 60A and 60A. 60B images. In addition, the pedestrian 200 is equipped with an acceleration measuring device 50 on the body, and the acceleration measuring device 50 measures the acceleration generated during walking of the pedestrian 200.

  (1) The omnidirectional image capturing apparatus 60 </ b> A transmits image data A generated by capturing the pedestrian 200 to the server 10. Similarly, the omnidirectional image capturing device 60 </ b> B transmits image data B generated by capturing the pedestrian 200 to the server 10.

  In addition, the acceleration measuring device 50 transmits measurement data generated by measuring acceleration generated during walking of the pedestrian 200 to the server 10.

  The imaging start time of image data A and image data B and the measurement start time of measurement data are synchronized, and the imaging end time of image data A and image data B and the measurement end time of measurement data are synchronized. Yes.

  (2) The server 10 synchronizes the time axes of the image data A and the measurement data, and synchronizes the time axes of the image data B and the measurement data. And the server 10 produces the display image data which displays either the image data A or the image data B according to time.

  (3) The server 10 transmits display image data and measurement data to the display device 30. Thereby, the display device 30 can display the display image data and the measurement data on the same time axis. The display image data is, for example, image data that displays which of the image data A and the image data B the pedestrian 200 is shown larger.

  According to the configuration shown in FIG. 1, two omnidirectional image capturing devices 60A and 60B are used, and the pedestrian 200 walking around the pedestrian 200 is imaged. Therefore, the pedestrian 200 moving linearly is used. In addition, it is possible to image the pedestrian 200 that turns, turns right, turns left, and the like. It is known that the pedestrian 200 easily loses its center of gravity when turning, turning right, turning left, or the like. For this reason, in the image display system 100 according to the present embodiment, in addition to the case of linear movement, it is possible to effectively analyze the walking motion of the pedestrian 200 that makes a turn, right turn, left turn, or the like.

  Further, since the omnidirectional image capturing devices 60A and 60B can capture the side surface of the pedestrian 200 without moving the camera, it is possible to capture how the body of the pedestrian 200 shakes in the front-rear direction. For this reason, according to the image display system 100 according to the present embodiment, an instructor or the like of walking can check and evaluate the walking motion of the pedestrian 200 with an image, and can guide improvement points. .

  In the image display system 100 according to the present embodiment, the measurement data obtained by measuring the acceleration or the like of the pedestrian 200 is displayed on the same time axis as the display image data. For this reason, when a leader etc. discover the walking motion which should be improved, it can be proved by measurement data that the walking motion is not good.

  In addition, in the image display system 100 according to the present embodiment, a leader or the like finds measurement data that is not good by comparing the measurement data that is prepared in advance with good measurement data and the measurement data of the pedestrian 200. You can also Since the measurement data that is not good is associated with the display image data, the instructor or the like can specify the walking motion to be improved from the display image data.

  Therefore, by using the image display system 100 according to the present embodiment, a leader or the like can evaluate the walking motion using the measurement data and the display image data in a complementary manner.

  Hereinafter, when the omnidirectional image capturing devices 60A and 60B are not distinguished from each other, they are simply expressed as “omnidirectional image capturing device 60”. When the image data A generated by the omnidirectional image capturing device 60A and the image data B generated by the omnidirectional image capturing device 60B are not distinguished from each other, simply “image data” is used. ". Further, in the following description, it is assumed that the image data is moving image data. Therefore, the image data includes a plurality of frame data that is data of an image (frame image) for one frame. However, the image data is not limited to moving images, and may be image data of still images (still image data).

  Here, the pedestrian 200 is an object that moves forward, backward, laterally, and the like by an action such as walking, but is not limited to a person, and may be, for example, an animal, a robot, or the like. Also, walking may include running or may walk backwards. Further, the object to be moved may be a wheelchair, a person wearing a prosthetic leg, a bicycle, a stilt, a tricycle, or the like. Hereinafter, for the sake of convenience, it is assumed that the moving object is “pedestrian 200”.

  The measurement data is information used to evaluate the movement of the moving object, but is not limited to acceleration, and may be, for example, angular velocity, position, or direction.

  The image data A and the measurement data are associated with each other when the one or more frame data A among the frame data A included in the image data A is determined. Means that one or more measured values are determined. Alternatively, when the frame data A included in the image data A captured at substantially the same time and the measurement value included in the measurement data are specified, the remaining time is determined when one of the time, the frame data A, or the measurement value is determined. Is determined. The same applies to image data B and display image data.

  The instructor refers to a person who can confirm the walking motion of the pedestrian 200 with an image, evaluate the walking motion, and plan improvement points. Specific examples include holders of qualifications such as public health nurses, clinical laboratory technicians, and prosthetic limb orthoses.

  Further, even if the person is not a person who can evaluate the walking motion and plan improvement points, the image display system 100 can be operated, and the instructor may be called an operator, a user, a user, or the like. .

<System configuration>
Next, the system configuration of the image display system 100 according to the present embodiment will be described. FIG. 2 is a diagram illustrating an example of a system configuration of the image display system 100 according to the present embodiment.

  An image display system 100 according to the present embodiment includes a server 10, a display device 30, an acceleration measuring device 50, and an omnidirectional image capturing device 60, which are connected to each other via a network N so as to communicate with each other.

  The network N is a LAN (Local Area Network) constructed in a facility where the image display system 100 is arranged. The network N may include the Internet, a telephone line network, and the like. Further, the network N may be constructed by either wired or wireless, and may be a combination of wired and wireless.

  The omnidirectional image capturing device 60 is an image capturing device (camera) that captures a 360-degree circumference called an omnidirectional image or an omnidirectional image (hereinafter simply referred to as an “omnidirectional image”). The omnidirectional image capturing device 60 includes an optical system to be described later, so that an omnidirectional image can be obtained by a single imaging operation. Since the omnidirectional image includes an image of 360 ° around the blind spot, an instructor or the like can selectively display an arbitrary region on a display or the like for viewing.

  In the present embodiment, the omnidirectional image capturing device 60 may not have an image capturing range of 360 ° in the latitude direction (the vertical direction of the image). This is because the pedestrian 200 is not shown in the area such as the ceiling or the floor.

  Further, the omnidirectional image capturing device 60 may not have an imaging range of 360 ° in the longitude direction (the horizontal direction of the image). This is because an image of the entire celestial sphere image capturing device 60 may not be necessary for the instructor to analyze the walking motion of the pedestrian 200.

  As the omnidirectional image capturing device 60, an image capturing device that obtains an omnidirectional image by connecting a plurality of images obtained by capturing a plurality of rectangular ranges at a certain angle of view may be used.

  The acceleration measuring device 50 is a device that measures the acceleration of the pedestrian 200 and obtains measurement data including a measured value indicating the measured acceleration. The acceleration is preferably measured on each of the three axes of vertical direction acceleration, traveling direction acceleration, and left-right direction acceleration, but may be acceleration in at least one direction.

  A dedicated sensor (acceleration sensor) for measuring acceleration also exists as the acceleration measuring device 50, but the acceleration measuring device 50 may be a device having an acceleration sensor. Specifically, the acceleration measuring device 50 is, for example, a smartphone, a mobile phone, a tablet terminal, a notebook PC (Personal Computer), a wearable terminal (a PC mounted as a head mounted display, sunglasses, a wristwatch, or the like), a pedometer, or the like. There may be.

  Further, the acceleration measuring device 50 may have a function of measuring, for example, angular velocity, position, azimuth, and the like in addition to acceleration.

  The angular velocity can be measured using, for example, a gyro sensor. The position can be measured using, for example, a GPS (Global Positioning System) satellite, IMES (Indoor MEssaging System), or a beacon. The azimuth can be measured, for example, using an azimuth sensor that detects the east, west, south, and north directions, for example, based on the north. Any of these can be used for evaluating the walking motion of the pedestrian 200 as well as the acceleration.

  In the present embodiment, the acceleration measurement device 50 is described as measuring acceleration. However, the present invention is not limited to this, and the acceleration measurement device 50 uses various measurement values used for evaluation of walking motion, such as angular velocity, position, and orientation. The included measurement data may be transmitted to the server 10.

  The server 10 is an information processing device that synchronizes the time axes of the image data received from the omnidirectional image capturing device 60 and the measurement data received from the acceleration measuring device 50. Further, the server 10 creates display image data from the image data received from the omnidirectional image capturing device 60. The function of the server 10 may be included in the acceleration measuring device 50 or the display device 30.

  The display device 30 is an information processing device used as a terminal device that allows a leader or the like to display display image data and measurement data. Specifically, for example, a PC, a smartphone, a tablet terminal, a mobile phone, a car navigation terminal, a wearable terminal, a camera, an electronic blackboard, a projector, a game machine, an MFP (Multifunction Peripheral / Printer / Product), and the like.

<Hardware configuration>
Next, the hardware configuration of the server 10, the display device 30, the acceleration measuring device 50, and the omnidirectional image capturing device 60 according to the present embodiment will be described.

  FIG. 3 is a diagram illustrating an example of a hardware configuration of the server 10 and the display device 30. Although the hardware configurations of the server 10 and the display device 30 illustrated in FIG. 3 are illustrated as examples arranged in one housing, the present invention is not limited thereto. The hardware configurations of the server 10 and the display device 30 may be distributed in, for example, a plurality of cases so that the respective hardware can communicate with each other. Since the server 10 and the display device 30 have the same hardware configuration, the hardware configuration of the server 10 will be mainly described below.

  The server 10 includes a central processing unit (CPU) 101, a read only memory (ROM) 102, a random access memory (RAM) 103, a hard disk drive (HDD) 105, and a media drive 107. The server 10 also includes a display 108, a network I / F 109, a keyboard 111, a mouse 112, and an optical drive 114. Each of these pieces of hardware is connected to each other by a bus 110.

  The CPU 101 executes an image processing program stored in an HD (Hard Disk) 104 and controls the overall operation of the server 10. The ROM 102 stores a program used to drive the CPU 101 such as an IPL (Initial Program Reader). The RAM 103 is a main storage device used as a work area for the CPU 101. The HD 104 is a storage device equipped with a nonvolatile memory.

  The HDD 105 controls reading or writing of various data with respect to the HD 104 according to the control of the CPU 101. The display 108 displays various information such as a cursor, menu, window, character, or image. The network I / F 109 is an interface with the network N. The display 108 of the server 10 may be the display device 30.

  The keyboard 111 and the mouse 112 are input / output devices. The keyboard 111 includes a plurality of keys for inputting characters, numerical values, various instructions, and the like, and receives input from the plurality of keys. The mouse 112 receives movement of the mouse pointer, selection and execution of various instructions, selection of a processing target, and the like.

  The media drive 107 controls reading or writing of various data with respect to the recording medium 106 such as a flash memory. The optical drive 114 controls reading or writing of various data with respect to a CD (Compact Disc) 113 as an example of a removable recording medium.

  Note that the above-described image processing program may be recorded and distributed on a computer-readable recording medium such as the recording medium 106 or the CD 113, for example, in a file in an installable format or an executable format. For example, the image processing program may be distributed in a form downloaded from a predetermined server device or the like.

  The server 10 and the display device 30 according to the present embodiment can implement various processes described later by having the hardware configuration illustrated in FIG. 3.

  FIG. 4 is a diagram illustrating an example of a hardware configuration of the acceleration measuring device 50. In addition, in FIG. 4, although the hardware configuration in case the acceleration measuring apparatus 50 is a smart phone or a tablet terminal is shown as an example, the hardware configuration of the acceleration measuring apparatus 50 is not restricted to this.

  The acceleration measuring apparatus 50 includes a CPU 201, ROM 202, RAM 203, flash memory 204, acceleration / direction sensor 205, gyro sensor 206, and media drive 207. The acceleration measuring device 50 includes a voice input device 210, a voice output device 211, a communication device 212, a GPS receiver 213, a display 214, and a touch panel 215. These pieces of hardware are connected to each other via a bus 209.

  The CPU 201 controls the overall operation of the acceleration measuring device 50 by executing a program stored in the flash memory 204. The ROM 202 stores IPL and basic input / output programs. A RAM 203 is a main storage device used as a work area when the CPU 201 executes a program.

  The flash memory 204 is a nonvolatile storage device that stores programs executed by the acceleration measuring device 50 and various data. The program is, for example, an operating system, a device program executed by the acceleration measuring device 50, or the like.

  The acceleration / orientation sensor 205 has a function as an electronic magnetic compass for detecting geomagnetism, a function for measuring acceleration in three axial directions, and the like. The gyro sensor 206 detects an angular velocity when the acceleration measuring device 50 rotates with respect to the x axis, the y axis, or the z axis. The rotation angle of each axis is called a yaw angle, a pitch angle, and a roll angle.

  The media drive 207 controls reading or writing (storage) of data with respect to a recording medium 208 such as a flash memory.

  The voice input device 210 is a microphone that converts voice into a voice signal. The audio output device 211 is a speaker that converts an audio signal into audio. The communication device 212 performs communication using a wireless communication signal using the antenna 212a. The communication device 212 may be a LAN card connected to the LAN.

  The GPS receiver 213 detects position information (for example, latitude, longitude, altitude, etc.) of the acceleration measuring device 50 by a GPS satellite or IMES as indoor GPS.

  The display 214 displays various information such as a cursor, menu, window, character, or image. The touch panel 215 is superimposed on the display 214 and detects a touch position (coordinates) on the display 214 in response to an operation with a finger, a touch pen, or the like.

  Note that the above-described apparatus program may be recorded in a computer-readable recording medium such as the recording medium 208 and distributed, for example, in an installable or executable file. The device program may be distributed, for example, in a form downloaded from a predetermined server device or the like.

  The acceleration measuring apparatus 50 according to the present embodiment can realize various processes described later by having the hardware configuration shown in FIG.

  FIG. 5 is a diagram illustrating an example of a hardware configuration of the omnidirectional image capturing apparatus 60. In FIG. 5, as an example, the omnidirectional image pickup device 60 is described as an image pickup device using two image pickup devices, but the number of image pickup devices may be three or more. Further, the omnidirectional image capturing device 60 is not necessarily a dedicated device for capturing an omnidirectional image. By attaching a retrospective omnidirectional image capturing unit to an ordinary digital camera, a smartphone, or the like. The omnidirectional image pickup device 60 may have substantially the same function.

  The omnidirectional image capturing device 60 includes an image capturing unit 301, an image processing unit 304, an image capturing control unit 305, a microphone 308, a sound processing unit 309, an audio output device 319, and a CPU 311. The omnidirectional image capturing device 60 includes a ROM 312, an SRAM (Static RAM) 313, a DRAM (Dynamic RAM) 314, an operation device 315, an external I / F 316, and a communication device 317.

  The imaging unit 301 includes wide-angle lenses (so-called fish-eye lenses) 302a and 302b each having an angle of view of 180 ° or more for forming a hemispherical image, and two imaging elements 303a provided corresponding to the wide-angle lenses. And 303b.

  The image sensors 303a and 303b each convert an optical image obtained by the corresponding wide-angle lens 302a or 302b into image data of an electrical signal and output it, and a timing for generating a horizontal or vertical synchronization signal, a pixel clock, and the like of the image sensor. A generation circuit and the like. The image sensors 303a and 303b include a register group in which various commands, parameters, and the like necessary for the operation of these image sensors are set. Examples of the image sensor include a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor.

  The imaging elements 303a and 303b of the imaging unit 301 are connected to the image processing unit 304 by a parallel I / F bus. In addition, the imaging elements 303a and 303b of the imaging unit 301 are connected to the imaging control unit 305 by a serial I / F bus (I2C bus or the like).

  The image processing unit 304 and the imaging control unit 305 are connected to the CPU 311 via the bus 310. In addition, a ROM 312, an SRAM 313, a DRAM 314, an operation device 315, an external I / F 316, a communication device 317, an electronic compass 318, and the like are connected to the bus 310.

  The image processing unit 304 takes in the image data output from the image sensors 303a and 303b through the parallel I / F bus, performs predetermined processing on each image data, and then combines these image data. Create image data of equirectangular images.

  In general, the imaging control unit 305 sets a command or the like in a register group of the imaging elements 303a and 303b using the I2C bus with the imaging control unit 305 as a master device and the imaging elements 303a and 303b as slave devices. At this time, necessary commands and the like are received from the CPU 311. In addition, the imaging control unit 305 also uses the I2C bus to capture the status data of the register groups of the imaging elements 303a and 303b and send it to the CPU 311.

  The imaging control unit 305 instructs the imaging elements 303a and 303b to output image data at the timing when the shutter button of the operation device 315 is pressed. Some omnidirectional image capturing devices 60 may have a preview display function by a display or a function corresponding to a moving image display. In this case, output of image data from the image sensors 303a and 303b is continuously performed at a predetermined frame rate.

  In addition, the imaging control unit 305 performs control to synchronize the output timing of the image data of the imaging elements 303a and 303b in cooperation with the CPU 311. In the present embodiment, the omnidirectional imaging device 60 is not provided with a display device, but may be provided with a display device.

  The microphone 308 converts sound into sound data. The sound processing unit 309 takes in the sound data output from the microphone 308 through the I / F bus, performs predetermined processing on the sound data, and creates an audio signal. The audio output device 319 is a speaker that converts an audio signal into audio.

  The CPU 311 controls the overall operation of the omnidirectional image capturing apparatus 60 and executes necessary processing. The ROM 312 stores various programs for the CPU 311. The SRAM 313 and the DRAM 314 are work memories, and store programs executed by the CPU 311 and data being processed. In particular, the DRAM 314 stores image data being processed by the image processing unit 304 and image data of an image created by a processed equirectangular projection.

  The operation device 315 is various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, and the like. The user can input various imaging modes, imaging conditions, and the like by operating the operation buttons.

  The external I / F 316 is an interface (for example, USB I / F) with an external medium such as an SD card or a PC. Further, the external I / F 316 may be a network interface regardless of wireless or wired. The image data of the image created by the equirectangular projection stored in the DRAM 314 is recorded on an external medium via the external I / F 316, or an external device (via the external I / F 316 as necessary). For example, it may be transmitted to a PC or the like.

  The communication device 317 communicates with an external device by a wireless technology such as Wi-Fi (Wireless Fidelity), NFC, or LTE (Long Term Evolution) via an antenna 317 a provided in the omnidirectional image capturing device 60. . Image data of an image created by equirectangular projection may be transmitted to an external device via the communication device 317.

  The electronic compass 318 calculates the azimuth and tilt (Roll rotation angle) of the omnidirectional imaging device 60 from the earth's magnetism, and outputs azimuth / tilt information. This azimuth / tilt information is related information (metadata) along Exif, and is used for image processing such as image correction of a captured image, for example. The related information includes the date and time of image capturing, the data capacity of image data, and the like.

  The omnidirectional image capturing apparatus 60 according to the present embodiment has the hardware configuration shown in FIG. 5, and can implement various processes described later.

<Functional configuration>
Next, functional configurations of the server 10, the display device 30, the acceleration measuring device 50, and the omnidirectional image capturing device 60 will be described. FIG. 6 is a diagram illustrating an example of functional blocks of the server 10, the display device 30, the acceleration measuring device 50, and the omnidirectional image capturing device 60.

<< Functional configuration of server 10 >>
The server 10 includes a communication unit 11, an image adjustment unit 12, an acceleration data processing unit 13, an operation input reception unit 14, a synchronization instruction unit 15, a display image creation unit 16, and a storage / readout unit 19. Each of these units is realized by any one of the constituent elements shown in FIG. 3 operating according to an instruction from the CPU 101 according to the image processing program 1010 expanded on the RAM 103.

  Further, the server 10 includes a storage unit 1000 constructed by any one or more of the HD 104, the ROM 102, and the RAM 103 shown in FIG. The storage unit 1000 stores an image processing program 1010, a measurement data DB 1001, and an image data DB 1002. The measurement data DB 1001 stores measurement data transmitted from the acceleration measurement device 50. The image data DB 1002 stores image data transmitted from the omnidirectional image capturing device 60.

  The communication unit 11 is realized by a command from the CPU 101, a network I / F 109, and the like, and transmits and receives various data to and from the acceleration measuring device 50, the omnidirectional image capturing device 60, and the display device 30 via the network N. .

  Hereinafter, even when the server 10 communicates with the function of the communication unit 11, the description “communication via the communication unit 11” may be omitted. The same applies to the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50.

  The image adjustment unit 12 is realized by an instruction from the CPU 101 and the like, and synchronizes the time axes of the image data acquired from the omnidirectional image capturing device 60 and the measurement data acquired from the acceleration measuring device 50.

  The acceleration data processing unit 13 is realized by an instruction from the CPU 101, and converts discrete measurement data into data represented by a continuous graph with respect to the time axis by, for example, smooth coupling.

  The operation input receiving unit 14 is realized by a command from the CPU 101, a keyboard 111, a mouse 112, and the like, and receives various operations and inputs to the server 10 by a leader or the like.

  The synchronization instructing unit 15 is realized by an instruction from the CPU 101, and performs processing related to synchronization between measurement data and image data. Specifically, the synchronization instruction unit 15 requests the acceleration measurement device 50 and the omnidirectional image imaging device 60 to start measurement and imaging at the same time. Similarly, the synchronization instruction unit 15 requests the acceleration measurement device 50 and the omnidirectional image imaging device 60 to end measurement and imaging at the same time, respectively. The synchronization instruction unit 15 provides absolute time for synchronization to the omnidirectional image capturing device 60 and the acceleration measuring device 50.

  The display image creation unit 16 is realized by an instruction from the CPU 101 and creates display image data based on the image data A and the image data B received from the omnidirectional image capturing devices 60A and 60B. For example, the display image creation unit 16 compares the frame data A included in the image data A at the same time with the frame data B included in the image data B, and the pedestrian 200 is shown larger in the frame image. Display image data including frame data is created.

  The storage / reading unit 19 is realized by an instruction from the CPU 101, the HDD 105, and the like, and performs processing for storing various data in the storage unit 1000 and reading various data stored in the storage unit 1000.

  Hereinafter, even when the server 10 accesses the storage unit 1000, the description “read / write via the storage / reading unit 19” may be omitted. The same applies to the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50.

<< Functional Configuration of Display Device 30 >>
The display device 30 includes a communication unit 31, a display control unit 32, an operation input reception unit 33, an image rotation unit 34, and a storage / readout unit 39. Each of these units is realized by any one of the components shown in FIG. 3 operating according to a command from the CPU 101 according to the terminal program 3010 expanded on the RAM 103. These units are not limited to the terminal program 3010, and may be realized by, for example, a Web page (HTML (HyperText Markup Language), XML (Extensible Markup Language), JavaScript (registered trademark), etc.) transmitted by the server 10. .

  In addition, the display device 30 includes a storage unit 3000 configured by any one or more of the HD 104, the ROM 102, and the RAM 103 illustrated in FIG. The storage unit 3000 stores a terminal program 3010. The storage unit 3000 stores measurement data and display image data transmitted from the server 10.

  The communication unit 31 is realized by a command from the CPU 101, a network I / F 109, and the like, and transmits and receives various data to and from the server 10 via the network N.

  The display control unit 32 is realized by an instruction from the CPU 101 and the like, and displays the display image data and measurement data received from the server 10 on the display 108.

  The operation input receiving unit 33 is realized by a command from the CPU 101, a keyboard 111, a mouse 112, and the like, and receives various operations and inputs to the display device 30 by a leader or the like.

  The image rotation unit 34 is realized by a command from the CPU 101 or the like, and rotates display image data that is an omnidirectional image in the horizontal direction so that an area where the pedestrian 200 is reflected is displayed on the display 108 (or Trimming).

  The storage / reading unit 39 is realized by an instruction from the CPU 101, the HDD 105, and the like, and performs a process of storing various data in the storage unit 3000 and reading out various data stored in the storage unit 3000.

≪Functional configuration of acceleration measuring device 50≫
The acceleration measuring device 50 includes a communication unit 51, an acceleration detection unit 52, a sound output unit 53, and a storage / readout unit 59. Each of these units is realized by any one of the constituent elements shown in FIG. 4 operating according to a command from the CPU 201 according to the device program 5010 expanded on the RAM 203.

  Further, the acceleration measuring apparatus 50 includes a storage unit 5000 configured by any one or more of the flash memory 204, the ROM 202, and the RAM 203 illustrated in FIG. The storage unit 5000 stores an apparatus program 5010. Further, the storage unit 5000 stores measurement data including measurement values such as acceleration measured by the acceleration measuring device 50.

  The communication unit 51 is realized by a command from the CPU 201, the communication device 212, and the like, and transmits and receives various data to and from the server 10 via the network N.

  The acceleration detection unit 52 is realized by a command from the CPU 201 and the acceleration / direction sensor 205 and the like, and measures the acceleration generated in the acceleration measuring device 50. The acceleration is measured at regular intervals or irregularly, and an acceleration that changes with time is obtained as a measurement value. Then, the acceleration detection unit 52 stores measurement data including measurement values obtained in time series in the storage unit 5000.

  The sound output unit 53 is realized by a command from the CPU 201, the sound output device 211, and the like, and outputs a sound such as a buzzer sound.

  The storage / reading unit 59 is realized by an instruction from the CPU 201, the flash memory 204, the ROM 202, the RAM 203, and the like, and stores various data in the storage unit 5000 and reads out various data stored in the storage unit 5000. I do.

≪Functional configuration of omnidirectional imaging device 60≫
The omnidirectional image capturing apparatus 60 includes a communication unit 61, an image capturing unit 62, a sound output unit 63, and a storage / readout unit 69. Each of these units is realized by any one of the constituent elements illustrated in FIG. 5 operating according to a command from the CPU 311 according to the imaging program 6010 expanded in the SRAM 313 or the DRAM 314.

  Further, the omnidirectional image capturing device 60 includes a storage unit 6000 constructed by one or more of the ROM 312, the SRAM 313, and the DRAM 314 shown in FIG. The storage unit 6000 stores an imaging program 6010. The storage unit 6000 stores image data captured by the omnidirectional image capturing device 60.

  The communication unit 61 is realized by an instruction from the CPU 311, a communication device 317, an external I / F 316, and the like, and transmits / receives various data to / from the server 10 via the network N.

  The imaging unit 62 is realized by a command from the CPU 311, the imaging unit 301, the image processing unit 304, and the like, and captures an omnidirectional image and generates image data. That is, the imaging unit 62 generates image data including frame data for each imaging interval, for example, by capturing images at regular intervals or irregularly.

  The sound output unit 63 is realized by a command from the CPU 311, a sound output device 319, and the like, and outputs a sound such as a buzzer sound.

  In the present embodiment, as described above, the image data is described as moving image data, but is not limited thereto, and may be still image data. In this case, for example, the imaging unit 62 may generate a plurality of image data by capturing images at predetermined time intervals while the pedestrian 200 is walking.

  Then, the imaging unit 62 stores image data including time-series frame data in the storage unit 6000. Note that the imaging unit 62 stores a plurality of time-series image data in the storage unit 6000 when image data of a still image is generated.

  The storage / reading unit 69 is realized by any one or more of instructions from the CPU 311, ROM 312, SRAM 313, and DRAM 314. The storage / reading unit 69 stores various data in the storage unit 6000 and reads various data stored in the storage unit 6000. Perform the process.

<About walking of pedestrian 200>
Here, how the pedestrian 200 walks around the omnidirectional imaging device 60 will be described. FIG. 7 is a diagram for explaining an example of walking of the pedestrian 200.

  As shown in FIG. 7, the pedestrian 200 wears the acceleration measuring device 50 on the waist or the like. The place where the acceleration measuring device 50 is attached is preferably a place close to the center of gravity of the pedestrian 200, but may be attached to the chest, back, torso, etc. in addition to the waist. The pedestrian 200 may be equipped with a plurality of acceleration measuring devices 50.

  The pedestrian 200 walks around the omnidirectional image pickup devices 60A and 60B along a line L (an 8-shaped line L) including a right turn portion, a left turn portion, and a straight portion. To do. Thus, when the pedestrian 200 walks along the line L, the instructor or the like walks when the pedestrian 200 turns to the right and walks, and walks when the pedestrian 200 turns to the left. It is possible to efficiently evaluate the movement and the walking movement when the vehicle is traveling straight.

  Note that a marker or the like may be arranged on the line L in order to make it easy to compare walking motions between the plurality of pedestrians 200 or to recognize a route on which the pedestrian 200 itself walks. Thereby, a leader etc. can grasp | ascertain easily the position of the pedestrian 200 with respect to the omnidirectional image imaging device 60, when confirming display image data.

  Note that while the pedestrian 200 is walking along the line L, the acceleration measuring device 50 measures the acceleration of the pedestrian 200. Assuming that the relative position of the pedestrian 200 and the acceleration measuring device 50 is fixed, the traveling direction (front) of the pedestrian 200 is the x direction, the left and right direction is the y direction, and the vertical direction is the z direction.

  Moreover, the pedestrian 200 does not have to walk the whole figure-shaped line L, and should just walk more than the distance suitable for evaluation of walking motion. That is, the pedestrian 200 should just walk either one about the two linear parts contained in the line L, for example. The pedestrian 200 may go around the figure-shaped line L a predetermined number of times.

<Operation procedure>
Next, an operation procedure of the image display system 100 according to the present embodiment will be described.

≪Overall operation≫
FIG. 8 is a sequence diagram illustrating an example of the overall operation of the image display system 100. FIG. 8 is a sequence diagram when the server 10 issues a measurement start request and an imaging start request to the acceleration measuring device 50 and the omnidirectional image capturing device 60, respectively.

  First, when starting measurement of walking motion, a leader or the like inputs an operation for starting measurement of walking motion to the server 10. Then, the operation input reception part 14 of the server 10 receives operation of a leader etc. (step S101).

  Next, the synchronization instruction unit 15 of the server 10 transmits a measurement start request to the acceleration measuring device 50 via the communication unit 11 when the operation input receiving unit 14 receives an operation of a leader or the like (step S102).

  When the communication unit 51 receives the measurement start request, the sound output unit 53 of the acceleration measuring device 50 outputs a buzzer sound or the like (step S103). Thereby, the pedestrian 200 can know that a walk may be started.

  The acceleration measuring device 50 notifies the pedestrian 200 of the start of walking by, for example, a voice message such as “Please start walking”, vibration by the vibration function, display of a message on the display 214, blinking of the display 214, or the like. May be.

  Next, the synchronization instruction unit 15 of the server 10 transmits an imaging start request to the omnidirectional image capturing devices 60A and 60B via the communication unit 11 (steps S104 and S105).

  When the communication unit 61 receives the imaging start request, the sound output unit 63 of the omnidirectional image capturing devices 60A and 60B outputs a buzzer sound or the like (steps S106 and S107). Thereby, similarly to the above, the pedestrian 200 can know that the walking may be started.

  Note that the omnidirectional image capturing device 60 may notify the pedestrian 200 of the start of walking by a voice message, vibration, or the like, similar to step S103 described above.

  Further, since the omnidirectional image capturing device 60 may enter the field of view of the pedestrian 200, for example, a patrol lamp or the like may be turned on. Thereby, not only the pedestrian 200 but also the surrounding leaders can know that the pedestrian 200 may start walking.

  Note that any one of notifications by the buzzer in steps S103, S106, and S107 may be performed.

  Next, the acceleration detector 52 of the acceleration measuring device 50 starts measuring acceleration (step S108). Then, the acceleration detection unit 52 of the acceleration measuring device 50 measures acceleration generated while the pedestrian 200 is walking, and stores measurement data including measurement values obtained in time series in the storage unit 5000.

  In addition, the imaging units 62 of the omnidirectional imaging devices 60A and 60B start imaging (steps S109 and S110), respectively. Then, the imaging units 62 of the omnidirectional image imaging devices 60A and 60B store the image data A and the image data B obtained by imaging the walking pedestrian 200 in the storage unit 6000, respectively.

  As described above, the acceleration measuring device 50 and the omnidirectional image capturing device 60 can synchronize the measurement data measurement start and the image data imaging start almost simultaneously with an instruction from the server 10.

  In addition, the order of step S102, step S104, and step S105 is random. The server 10 may transmit the measurement start request and the imaging start request to the acceleration measuring device 50 and the omnidirectional image capturing device 60, respectively, at the same time.

  Next, for example, when a leader or the like determines that the pedestrian 200 has sufficiently walked, the leader or the like inputs an operation to end the measurement of the walking motion to the server 10. Then, the operation input reception part 14 of the server 10 receives operation of a leader etc. (step S111).

  Next, when the operation input accepting unit 14 accepts an operation of a leader or the like, the synchronization instruction unit 15 of the server 10 transmits a measurement end request to the acceleration measuring device 50 via the communication unit 11 (step S112). And the acceleration detection part 52 of the acceleration measuring apparatus 50 will complete | finish the measurement of an acceleration, if the communication end part 51 receives a measurement end request | requirement. Note that, similarly to step S103, the end of measurement may be notified to the pedestrian 200 or the like with a buzzer or the like.

  Next, the synchronization instruction unit 15 of the server 10 transmits an imaging end request to the omnidirectional image capturing devices 60A and 60B via the communication unit 11 (steps S113 and S114). Then, when the imaging unit 62 of the omnidirectional image imaging device 60 receives the imaging end request through the communication unit 61, the imaging unit 62 ends imaging. Note that, similarly to step S106 and step S107, the end of imaging may be notified to the pedestrian 200 or the like with a buzzer or the like.

  In this way, the acceleration measuring device 50 and the omnidirectional image capturing device 60 can synchronize the measurement data measurement end and the image data image capturing end almost simultaneously in response to an instruction from the server 10.

  In addition, the order of step S112, step S113, and step S114 is random. The server 10 may transmit a measurement end request and an imaging end request to the acceleration measuring device 50 and the omnidirectional image capturing device 60, respectively, at the same time.

  Next, the communication unit 51 of the acceleration measuring device 50 transmits the measurement data stored in the storage unit 5000 to the server 10 (step S115). Further, the communication units 61 of the omnidirectional image capturing apparatuses 60A and 60B transmit the image data A and the image B stored in the storage unit 6000 to the server 10 (steps S116 and S117).

  Next, when the acceleration data processing unit 13 of the server 10 receives the measurement data, the image data A, and the image data B by the communication unit 11, the measurement data is stored in the measurement data DB 1001, and the image data A and the image data B are stored in the image data. Each of them is stored in the DB 1002. Then, the acceleration data processing unit 13 processes the measurement data, which is discrete data, into continuous data represented by, for example, a line graph (step S118). Such processing may be performed by the display device 30.

  Next, the image adjustment unit 12 of the server 10 performs a process of synchronizing the time axes of the measurement data and the image data A and the time axes of the measurement data and the image data B (step S119). That is, the image adjustment unit 12 associates each measurement value included in the measurement data with each frame data A included in the image data A. Similarly, the image adjustment unit 12 associates each measurement value included in the measurement data with each frame data B included in the image data B.

  Here, the measurement data and image data are acquired at almost the same time, but the number of measurement data (number of measurement values) is different from the number of image data (number of frame data). There are many. For this reason, the image adjustment unit 12 synchronizes the time axis by associating the measurement data with the image data. Since the measurement start time and the imaging start time are substantially the same, and the measurement end time and the imaging end time are substantially the same, the image adjustment unit 12 can synchronize the time axis with the ratio of the number of data of the measurement data and the image data. it can.

  For example, when the number of measurement data is 200 and the number of image data is 400, two frame data included in the image data may be associated with one measurement value included in the measurement data. Alternatively, the ratio of the measurement time of each measurement value with respect to the measurement time and the ratio of the imaging time of each frame data with respect to the imaging time may be calculated, and the frame data with the closest ratio may be associated with the measurement value. Alternatively, a measurement value and frame data having substantially the same elapsed time from the start of measurement and the elapsed time from the start of imaging may be associated with each other.

  As will be described later, when the absolute time of the measurement start time, the measurement end time, the imaging start time, and the imaging end time is obtained, the image adjustment unit 12 performs measurement data and image data as follows. May be associated with each other. That is, the image adjustment unit 12 extracts the measurement value and the frame data in the overlapping portion between the time from the measurement start time to the measurement end time and the time from the imaging start time to the imaging end time, and calculates the ratio as described above. It is sufficient to associate with each other.

  Next, the display image creation unit 16 of the server 10 creates display image data based on the image data A and the image data B (step S120). That is, for example, the display image creation unit 16 acquires frame data A included in the image data A and frame data B included in the image data B at the same time. Then, the display image creation unit 16 compares the frame image A displayed by the acquired frame data A and the frame image B displayed by the frame data B, and the one where the pedestrian 200 is shown larger. Display image data is created so as to include frame data.

  Thereby, it is possible to cause the display device 30 to display an image in which the pedestrian 200 is larger in the images captured by the omnidirectional image capturing device 60A and the omnidirectional image capturing device 60B. It becomes like this. For this reason, it becomes easy for a leader etc. to confirm walk operation of pedestrian 200. Details of the processing in step S120 will be described later.

  Next, the communication unit 11 of the server 10 transmits the measurement data and the display image data to the display device 30 (Steps S121 and S122).

  Specifically, for example, a leader or the like requests measurement data and display image data from the server 10 via the display device 30 or the like. Or a leader etc. may perform operation which requests | requires measurement data and display image data with respect to the server 10, and the operation input reception part 14 of the server 10 may receive this operation. Alternatively, the display device 30 may inquire about the presence of measurement data and display image data by polling or the like, and the server 10 may transmit the measurement data and display image data to the display device 30 in response to the inquiry.

  Next, when the measurement data and the display image data are received by the communication unit 31, the display control unit 32 of the display device 30 displays the measurement data and the display image data on the display 108 on the same time axis (step S123). A process for displaying the measurement data and the display image data will be described later.

≪Modification of overall operation (part 1) ≫
Here, a modified example (part 1) of the overall operation of the image display system 100 will be described. FIG. 9 is a sequence diagram illustrating a modification example (No. 1) of the overall operation of the image display system 100. In the modified example (part 1) shown in FIG. 9, the trigger for starting measurement is different from that in FIG. Note that step S203 to step S211 and step S214 to step S225 in FIG. 9 are the same as step S102 to step S110 and step S112 to step S123 in FIG.

  First, when starting measurement of walking motion, a leader or the like inputs an operation for starting measurement of walking motion to the display device 30. Then, the operation input receiving unit 33 of the display device 30 receives an operation of a leader or the like (step S201).

  Next, the communication part 31 of the display apparatus 30 will transmit a measurement start request | requirement to the server 10, if operation of a leader etc. is received by the operation input reception part 33 (step S202). The display device 30 can communicate with the server 10 by, for example, HTTP (Hypertext Transfer Protocol). The measurement start request is a request for at least one of acceleration measurement start and pedestrian 200 imaging start.

  For example, when the leader or the like determines that the pedestrian 200 has sufficiently walked, the leader or the like inputs an operation to end the measurement of the walking motion to the display device 30. Then, the operation input receiving unit 33 of the display device 30 receives an operation of a leader or the like (step S212).

  Next, the communication part 31 of the display apparatus 30 will transmit a measurement end request | requirement to the server 10, if operation of a leader etc. is received by the operation input reception part 33 (step S213). The measurement end request is a request for at least one of the acceleration measurement end and the pedestrian 200 imaging end.

  As described above, in the modified example (No. 1) of the overall operation illustrated in FIG. 9, the leader or the like requests the server 10 to start and end the measurement from the display device 30 without operating the server 10. Can do. For this reason, when a leader or the like is operating the display device 30, the leader or the like can start measurement without having to move to the server 10.

<< Modified example of overall operation (2) >>
Next, a modification (No. 2) of the overall operation of the image display system 100 will be described. FIG. 10 is a sequence diagram illustrating a modification (No. 2) of the overall operation of the image display system 100. In the modification (No. 2) shown in FIG. 10, the start of measurement is triggered by the start of walking by the pedestrian 200. The pedestrian 200 starts and ends walking, for example, according to instructions from a leader or the like. Alternatively, walking is started by starting walking from the start position on the line L, going around the line L and returning to the starting position. Note that steps S306 to S310 and steps S315 to 323 in FIG. 10 are the same as steps S108 to S110 and steps S115 to S123 in FIG.

  First, the acceleration detector 52 of the acceleration measuring device 50 detects that the pedestrian 200 has started walking based on the acceleration (step S301).

  For example, the acceleration detection unit 52 can detect that the pedestrian 200 has started walking by detecting an acceleration equal to or greater than a threshold value in one or more of the x direction, the y direction, and the z direction. In addition, for example, the acceleration detection unit 52 may detect that the pedestrian 200 has started walking because a change in position per unit time obtained by integrating acceleration twice is equal to or greater than a threshold value. . In addition to these, the acceleration detection unit 52 is configured so that the pedestrian 200 walks based on the angular velocity detected by the gyro sensor 206, the position detected by the GPS receiver 213, the direction detected by the acceleration / direction sensor 205, and the like. May be detected.

  And the acceleration detection part 52 of the acceleration measuring apparatus 50 starts the measurement of acceleration by detecting the start of the walk of the pedestrian 200.

  Next, the communication unit 51 of the acceleration measuring device 50 transmits a walking start notification to the server 10 (step S302). Further, the sound output unit 53 of the acceleration measuring device 50 outputs a buzzer sound or the like for notifying that the walking has started (step S303).

  Next, the synchronization instruction unit 15 of the server 10 transmits an imaging start request to the omnidirectional image capturing devices 60A and 60B via the communication unit 11 (steps S304 and S305). In addition, the acceleration measuring device 50 may transmit a walking start notification to the omnidirectional image capturing devices 60A and 60B as an imaging start request.

  The acceleration detector 52 of the acceleration measuring device 50 detects that the pedestrian 200 has stopped walking based on the acceleration (step S311).

  For example, the acceleration detection unit 52 can detect that the pedestrian 200 has stopped walking because no acceleration greater than or equal to the threshold is detected in one or more of the x direction, the y direction, and the z direction. Further, for example, the acceleration detection unit 52 may detect that the pedestrian 200 has stopped walking because the change in the position of the unit time obtained by integrating the acceleration twice becomes less than the threshold value. In addition to these, the acceleration detection unit 52 is configured so that the pedestrian 200 walks based on the angular velocity detected by the gyro sensor 206, the position detected by the GPS receiver 213, the direction detected by the acceleration / direction sensor 205, and the like. It may be detected that has stopped.

  And the acceleration detection part 52 complete | finishes the measurement of acceleration by detecting the stop of the walk of the pedestrian 200. FIG.

  The communication unit 51 of the acceleration measuring device 50 transmits a walking end notification to the server 10 (step S312).

  Next, the synchronization instruction unit 15 of the server 10 transmits an imaging end request to the omnidirectional image imaging devices 60A and 60B via the communication unit 11 (steps S313 and S314). In addition, the acceleration measuring device 50 may transmit a walking end notification to the omnidirectional image capturing devices 60A and 60B as an imaging end request.

  As described above, in the modified example (No. 2) of the overall movement shown in FIG. 10, the measurement of the walking movement may be started and ended without an instructor or the like performing an operation such as measurement start or measurement end. it can.

<< Modified example of overall operation (part 3) >>
Next, a modification (No. 3) of the overall operation of the image display system 100 will be described. FIG. 11 is a sequence diagram for explaining a modification (No. 3) of the overall operation of the image display system 100. In the modification (No. 3) shown in FIG. 11, the measurement data and the image data are synchronized using the absolute time. Note that steps S401 to S403, steps S405 to S408, and steps S411 to S415 in FIG. 11 are the same as steps S101 to S103, steps S104 to S107, and steps S108 to S112 in FIG. 8, respectively. Therefore, the description thereof is omitted. Further, steps S417 to S418 and steps S421 to S429 in FIG. 11 are the same as steps S113 to S114 and steps S115 to S123 in FIG.

  When receiving the measurement start request, the communication unit 51 of the acceleration measuring device 50 transmits a start time request to the server 10 (step S404). Then, the synchronization instruction unit 15 of the server 10 returns the absolute time (measurement start time) when the acceleration measuring device 50 starts measurement via the communication unit 11. Thereby, the acceleration measuring device 50 can obtain the measurement start time.

  Next, when the communication unit 61 of the omnidirectional image capturing apparatuses 60A and 60B receives the imaging start request, the communication unit 61 transmits a start time request to the server 10 (steps S409 and S410), respectively. Then, the synchronization instruction unit 15 of the server 10 returns the absolute time (imaging start time) when the omnidirectional image capturing devices 60A and 60B start capturing via the communication unit 11. Thereby, the omnidirectional image capturing devices 60A and 60B can obtain the imaging start time, respectively.

  When receiving the measurement end request, the communication unit 51 of the acceleration measuring device 50 transmits an end time request to the server 10 (step S416). Then, the synchronization instruction unit 15 of the server 10 returns an absolute time (measurement end time) when the acceleration measuring device 50 ends the measurement via the communication unit 11. Thereby, the acceleration measuring apparatus 50 can obtain the measurement end time.

  When receiving the imaging end request, the communication unit 61 of the omnidirectional image capturing apparatuses 60A and 60B transmits an end time request to the server 10 (steps S419 and S420). Then, the synchronization instruction unit 15 of the server 10 returns the absolute time (imaging end time) when the omnidirectional image capturing devices 60 </ b> A and 60 </ b> B end capturing via the communication unit 11. As a result, the omnidirectional image capturing devices 60A and 60B can obtain the imaging end time.

  As described above, in the modified example (No. 3) of the overall operation illustrated in FIG. 11, the server 10 returns the measurement start time and the measurement end time to the acceleration measuring device 50, and sets the imaging start time and the imaging end time. It returns to the omnidirectional image capturing device 60. For this reason, in the modification (No. 3), the server 10 associates the measurement data with the image data based on the absolute time in step S424. Therefore, for example, even if a time lag or the like occurs between the start of acceleration measurement and the start of imaging for some reason, or between the end of measurement of acceleration and the end of imaging, the server 10 Can be synchronized.

<< Modified example of overall operation (Part 4) >>
Next, a modification (No. 4) of the overall operation of the image display system 100 will be described. FIG. 12 is a sequence diagram illustrating a modification (No. 4) of the overall operation of the image display system 100. In the modification (No. 4) shown in FIG. 12, the acceleration measuring device 50 transmits an imaging start request to the omnidirectional image capturing device 60, and the absolute time is set according to the request from the omnidirectional image capturing device 60. I will reply. Note that steps S501 to S502 and step S508 in FIG. 12 are the same as steps S101 to S102 and step S103 in FIG. 12 are the same as Step S106 to Step S112 and Step S115 to Step S123 of FIG. 8, respectively, and thus description thereof is omitted.

  When receiving the measurement start request, the communication unit 51 of the acceleration measuring device 50 transmits the imaging start request to the omnidirectional image capturing devices 60A and 60B, respectively (steps S503 and S504).

  When receiving the imaging start request, the communication unit 61 of the omnidirectional image imaging devices 60A and 60B transmits a start time request to the acceleration measuring device 50, respectively (step S505 and step S506).

  Next, when receiving the start time request, the communication unit 51 of the acceleration measuring device 50 transmits the start time request to the server 10 (step S507). Then, the synchronization instruction unit 15 of the server 10 returns the measurement start time via the communication unit 11. And the communication part 51 of the acceleration measuring apparatus 50 will return an imaging start time to the omnidirectional image imaging devices 60A and 60B, respectively, if a measurement start time is received.

  The acceleration measuring device 50 may return the same time as the measurement start time received from the server 10 to the omnidirectional image imaging device 60 as the imaging start time.

  In step S505 and step S506, the time (current time) held by the acceleration measuring device 50 may be returned to the omnidirectional imaging device 60 as the imaging start time. The acceleration measuring device 50 starts measuring acceleration from the current time, and the omnidirectional imaging device 60 starts imaging from the imaging start time. Therefore, the measurement start time and the imaging start time can be synchronized. .

  When receiving the measurement end request, the communication unit 51 of the acceleration measuring device 50 transmits the imaging end request to the omnidirectional image capturing devices 60A and 60B, respectively (step S516 and step S517).

  When the communication unit 61 of the omnidirectional image capturing devices 60A and 60B receives the imaging end request, it transmits an end time request to the acceleration measuring device 50 (step S518 and step S519).

  Next, when receiving the end time request, the communication unit 51 of the acceleration measuring device 50 transmits the end time request to the server 10 (step S520). Then, the synchronization instruction unit 15 of the server 10 returns the measurement end time via the communication unit 11. Then, when receiving the measurement end time, the communication unit 51 of the acceleration measuring device 50 returns the imaging end time to the omnidirectional image capturing devices 60A and 60B, respectively.

  The acceleration measuring device 50 may return the same time as the measurement end time received from the server 10 to the omnidirectional image imaging device 60 as the imaging end time.

  In step S518 and step S519, the time (current time) held by the acceleration measuring device 50 may be returned to the omnidirectional imaging device 60 as the imaging end time. The acceleration measuring device 50 finishes measuring the acceleration at the current time, and the omnidirectional imaging device 60 finishes imaging at the imaging end time, so that the measurement end time and the imaging end time can be synchronized. .

<< Modified example of overall operation (part 5) >>
Next, a modification (No. 5) of the overall operation of the image display system 100 will be described. FIG. 13 is a sequence diagram for explaining a modification (No. 5) of the overall operation of the image display system 100. In the modified example (No. 5) illustrated in FIG. 13, one omnidirectional image imaging device 60 among the plurality of omnidirectional image imaging devices 60 starts imaging to the other omnidirectional image imaging devices 60. In addition to requesting, the acceleration measuring device 50 is requested to start measurement. Note that steps S601 and S605 in FIG. 13 are the same as steps S101 and 103 in FIG. Further, Steps S606 to S611 and Steps S618 to S623 in FIG. 13 are the same as Steps S106 to S111 and Steps S118 to S123 in FIG.

  The synchronization instruction unit 15 of the server 10 receives an operation of a leader or the like (operation for starting measurement of walking motion) by the operation input reception unit 14, and issues an imaging start request via the communication unit 11. The image is transmitted to the image capturing device 60A (step S602). The communication unit 11 transmits an imaging start request to one preset omnidirectional image capturing device 60 among the plurality of omnidirectional image capturing devices 60. Therefore, for example, the communication unit 11 may transmit an imaging start request to the omnidirectional image capturing device 60B. Hereinafter, a case where the communication unit 11 of the server 10 transmits an imaging start request to the omnidirectional image capturing device 60A will be described.

  Next, when receiving the imaging start request, the communication unit 61 of the omnidirectional image capturing apparatus 60A transmits a measurement start request to the acceleration measuring apparatus 50 and transmits an imaging start request to the omnidirectional image capturing apparatus 60B. (Step S603 and Step S604). As described above, when one omnidirectional image capturing device 60 among the plurality of omnidirectional image capturing devices 60 receives the imaging start request, the one omnidirectional image capturing device 60 transmits to the acceleration measuring device 50. A measurement start request is transmitted, and an imaging start request is transmitted to another omnidirectional image capturing apparatus 60.

  When the operation instruction accepting unit 14 accepts an operation of a leader or the like (an operation for ending the measurement of walking motion) by the operation input accepting unit 14, the synchronization instruction unit 15 of the server 10 sends an imaging end request via the communication unit 11. The image is transmitted to the image capturing device 60A (step S612).

  Next, when receiving the imaging end request, the communication unit 61 of the omnidirectional image capturing apparatus 60A transmits a measurement end request to the acceleration measuring apparatus 50 and transmits an imaging end request to the omnidirectional image capturing apparatus 60B. (Step S613 and Step S614). As described above, when one omnidirectional image capturing device 60 among the plurality of omnidirectional image capturing devices 60 receives the imaging end request, the one omnidirectional image capturing device 60 transmits the acceleration measuring device 50 to the acceleration measuring device 50. A measurement end request is transmitted, and an imaging end request is transmitted to the other omnidirectional imaging device 60.

  Next, the communication unit 51 of the acceleration measuring device 50 transmits the measurement data stored in the storage unit 5000 to the omnidirectional image capturing device 60A (step S615). In addition, the communication unit 61 of the omnidirectional image capturing apparatus 60B transmits the image data B stored in the storage unit 6000 to the omnidirectional image capturing apparatus 60A (step S616).

  Next, the communication unit 51 of the omnidirectional image capturing apparatus 60A receives the measurement data and the image data B received from the acceleration measuring apparatus 50 and the omnidirectional image capturing apparatus 60, respectively, as image data stored in the storage unit 6000. A together with A is transmitted to the server 10 (step S617).

  As described above, in the modified example (No. 5) of the overall operation illustrated in FIG. 13, one celestial sphere image capturing device 60 relays, and the acceleration measuring device 50 or another omnidirectional image capturing device 60. Send a start request to For this reason, for example, when the acceleration measuring device 50 and the other omnidirectional image capturing device 60 are not connected to the network N, the single omnidirectional image capturing device 60 uses Bluetooth (registered trademark) or the like. The start request can be transmitted using short-range wireless communication.

≪Display image creation process≫
Next, display image creation processing in step S120 of FIG. 8 will be described. FIG. 14 is a flowchart illustrating an example of the display image creation process. Hereafter, N frame data is included in the image data generated by the omnidirectional image capturing device 60 every time t = t _n (n = 1, 2,..., N). It shall be assumed. Note that Δt _n = t _n −t _n−1 (n = 2, 3,..., N) is an imaging interval.

First, the display image creation unit 16 sets time t = t _n (n = 2) (step S701).

  Next, the display image creation unit 16 acquires the frame data A and the frame data B at time t from the image data A and the image data B (step S702).

Next, the display image creation unit 16 captures the pedestrian 200 based on the difference between the frame data A and the frame data B from the previous frame data (that is, the frame data at time t = t _n−1 ). A rectangular area is identified (step S703).

  Here, the specification of the rectangular area in which the pedestrian 200 is shown will be described with reference to FIG. FIG. 15 is a diagram schematically illustrating an example of specifying a rectangular area in which a pedestrian 200 is shown.

First, in FIG. 15 (a), _t = t and _n frame image 181 frame data is displayed, t _{= t n-1} frame image 182 frame data is displayed (i.e., the previous frame image of the frame image 181 ) And are displayed. For example, when the frame rate is 30 fps, there is an interval of about 33 (milliseconds) between the frame image 181 and the immediately preceding frame image 182.

  The frame image 182 may be a frame image that is several frames before the frame image 181 instead of the frame image 181 and the immediately preceding frame image 182.

  The display image creation unit 16 calculates the difference between the frame image 181 and the immediately preceding frame image 182. The difference means that the pixel values of the pixels at the same position in the frame image 181 and the frame image 182 are subtracted. Note that even if the exact same subject is shown (even if there is no moving object), the difference may not be zero due to the blinking of the fluorescent light, the movement of a distant person, the change of the shadow, etc. Consider zero.

  FIG. 15B is an example of the difference image. In FIG. 15B, a pixel whose difference is greater than or equal to a threshold is replaced with a white pixel. Therefore, when the frame image 181 and the immediately preceding frame image 182 are compared, it can be said that there is a high possibility that the pedestrian 200 exists so as to overlap the white pixels.

  Next, as illustrated in FIG. 15C, the display image creation unit 16 performs pixel expansion processing on white pixels a plurality of times. The pixel expansion process is a process of replacing 8 pixels in contact with white pixels (or 4 pixels adjacent vertically and horizontally) with white pixels. Since the pixel in contact with the white pixel is traced by a plurality of pixel expansion processes, the white pixel appears to expand each time the number of times increases.

  The multiple times is a predetermined number such as 8 times or 10 times, and a suitable value is experimentally determined. Alternatively, the pixel expansion process may be stopped when the area of the white pixel exceeds a certain value.

  By executing the pixel expansion process a plurality of times, the white pixels that are scattered immediately after the difference between the frame image 181 and the immediately preceding frame image 182 is calculated become a block having a size. In the case of the pedestrian 200, since the difference image is vertically long, the white pixel is enlarged in the vertical direction by the pixel expansion process, and a slightly larger size than the moving object (a range slightly larger than the pedestrian 200) can be specified.

  Next, as shown in FIG. 15D, a circumscribed rectangle of a region where white pixels are continuous is specified as a rectangular region 601. Thereby, the rectangular area | region 601 in which the pedestrian 200 is reflected is specified. Note that there is a possibility that moving objects other than the pedestrian 200 exist around the omnidirectional image capturing device 60. Therefore, when a plurality of rectangular areas 601 are detected, for example, the area of the rectangular area 601 has an area. What is necessary is just to let the pedestrian 200 be reflected in the largest rectangular area 601.

Returning to FIG. The display image creation unit 16 sets the frame data having the larger rectangular area (ie, the larger area) specified in step S703 out of the frame data A and the frame data B as display frame data (step S704). ). That is, the display image creation unit 16 uses the frame data having the larger rectangular area specified in step S703 as frame data (display frame data) for displaying the frame image at time t = t _n in the display image data. To do.

Next, the display image creation unit 16 determines whether n is less than N (step S705). If n is less than N, the display image creation unit 16 adds 1 to n (step S706), and returns to step S702. In this way, the display image creation unit 16 displays the frame data in which the pedestrian 200 is greatly captured among the frame data A and the frame data B with respect to n = 2 to N, as the display frame at the time t = t _n . Data.

On the other hand, if n is not less than N, the display image creation unit 16 creates display image data including display frame data at each time t = t _n (n = 1, 2,..., N) (step S707). ). By creating display image data in this way, the display image creation unit 16 creates display image data using the frame data in which the pedestrian 200 is greatly photographed out of the frame data A and the frame data B as display frame data. can do. For n = 1, the display image creation unit 16 may use the frame data immediately before the frame data used as the display frame data when n = 2 as the display frame data.

  For example, when the display image data is played back by streaming or the like on the display device 30, the server 10 may transmit the display frame data to the display device 30 after the process of step S <b> 704 is performed.

≪Modification of display image creation process≫
Next, a modification of the display image creation process in step S120 of FIG. 8 will be described. FIG. 16 is a flowchart illustrating a modification of the display image creation process. In the modification of the display image creation process shown in FIG. 16, the display frame data is determined based on the size of the rectangular area of the pedestrian 200 in the immediately following frame data. Thereby, the display image creation unit 16 can display the image data captured by the omnidirectional image capturing device 60 in the direction in which the pedestrian 200 is approaching. In other words, the display image creation unit 16 can display image data in which the front of the pedestrian 200 is captured.

First, the display image creation unit 16 sets time t = t _n (n = 2) (step S801).

Next, the display image creation unit 16 acquires frame data A and frame data B at time t = t _n (n = 2) from each of the image data A and the image data B (step S802).

Next, the display image creation unit 16 captures the pedestrian 200 based on the difference between the frame data A and the frame data B from the previous frame data (that is, the frame data at time t = t _n−1 ). A rectangular area is identified (step S803). This can be done by specifying a rectangular area in the same way as in step S703 in FIG.

Next, the display image creation unit 16 obtains frame data having a larger rectangular area (that is, a larger area) identified in step S803 from the frame data A and the frame data B at time t = t _n. The display frame data is (n = 2) (step S804).

  Next, the display image creation unit 16 determines whether n is less than N (step S805). Then, when n is less than N, the display image creating unit 16 performs Steps S806 to S810.

That is, first, the display image creation unit 16 creates a rectangular area between frame data as display frame data and frame data immediately after the image data including the frame data (that is, frame data at time t = t _{n + 1} ). Compare (step S806). For example, when the display frame data at time t = t _n is frame data A, the display image creation unit shows the pedestrian 200 with the frame data A and the frame data A at time t = t _{n + 1} . Compare the size of the rectangular areas. Note that a rectangular area specifying method may be the same as that in step S703 in FIG.

Next, the display image creating unit 16 determines whether or not the rectangular area of the frame data at time t = t _{n + 1} is smaller (step S807). If the display image creation unit 16 determines that the rectangular area of the frame data at time t = t _{n + 1} is smaller, the display image creation unit 16 switches the image data and displays the frame data at time t = t _{n + 1} in the switched image data. Frame data is set (step S808).

For example, when the display frame data at time t = t _n is frame data A, the display image creation unit 16 switches from image data A to image data B, and the frame data B at time t = t _{n + 1} in the image data B. Is the display frame data at time t = t _{n + 1} .

On the other hand, if the display image creation unit 16 does not determine that the rectangular area of the frame data at time t = t _{n + 1} is smaller, the display image creation unit 16 sets the frame data as display frame data (step S809).

For example, when the display frame data at the time t = t _n is the frame data A, the display image creating unit 16 converts the frame data at the time t = t _{n + 1} in the image data A to the frame data at the time t = t _{n + 1} in the display image data. Time display frame data.

  As described above, the display image creation unit 16 determines whether or not the rectangular area of the pedestrian 200 has become smaller in the next frame data in the image data including the frame data as the display frame data. Then, the display image creation unit 16 switches the image data to other image data when the rectangular area becomes smaller. Thereby, when the pedestrian 200 is moving in a direction away from the omnidirectional image capturing device 60, it can be switched to other image data and displayed. In other words, the image data captured by the omnidirectional image capturing device 60 in the direction in which the pedestrian 200 approaches can be displayed.

  Next, the display image creation unit 16 adds 1 to n (step S810) and returns to step S805.

If n is not less than N in step S805, the display image creation unit 16 sets the frame data at time t = t _n in the image data including the frame data as display frame data as display frame data (step S811). .

  Then, the display image creation unit 16 creates display image data including display frame data (step S812). By creating display image data in this way, it is possible to create display image data in which the frame data in which the pedestrian 200 is facing the front of the frame data A and the frame data B is used as the display frame data. Therefore, for example, a leader or the like can easily confirm and evaluate the walking of the pedestrian 200.

  For n = 1, the display image creation unit 16 may use the frame data immediately before the frame data used as the display frame data when n = 2 as the display frame data.

In step S807 in FIG. 8, the display image creating unit 16 determines whether or not the rectangular area of the frame data at time t = t _{n + 1} is smaller, but determines whether or not the rectangular area is larger. You may judge. The display image creation unit 16 may use the frame data as display frame data when the rectangular area is larger.

<About other examples of walking of pedestrian 200>
Here, another example of walking of the pedestrian 200 will be described. FIG. 17 is a diagram for explaining another example of walking of the pedestrian 200.

  As shown in FIG. 17A, the pedestrian 200 may walk along the oval line L around the omnidirectional image capturing devices 60A and 60B. By such walking, a leader or the like can evaluate the walking motion of the walking in which the pedestrian 200 makes a large turn.

  As shown in FIG. 17B, the pedestrian 200 may walk along the two rhombus lines L around the omnidirectional image capturing devices 60A and 60B. By such walking, a leader or the like can evaluate the walking motion of walking in which the pedestrian 200 bends at a right angle (right turn or left turn).

<Display image data and measurement data>
Next, display examples of display image data and measurement data will be described. FIG. 18 is a diagram illustrating an example of display image data and measurement data displayed on the display device 30.

  A data display screen 501 shown in FIG. 18 mainly has an image data column 502 and a measurement data column 503. The display control unit 32 displays the display image data in the image data column 502 and displays the measurement data in the measurement data column 503.

  At this time, the display control unit 32 extracts the imaging range of the pedestrian 200 from the display image data and displays it in the image data field 502. This process will be described later.

  First, the image data column 502 will be described. When a leader or the like presses the button 504, the operation input receiving unit 33 receives the pressing operation. The display control unit 32 reproduces the display image data as a moving image at the same reproduction speed as the frame rate (fps). Note that high-speed playback, low-speed playback, and the like may be possible.

  When a leader or the like presses the button 504 again, the operation input receiving unit 33 receives the pressing operation. Then, the display control unit 32 stops the reproduction of the display image data. In this way, a leader or the like can display display image data at an arbitrary timing. Note that not only the display image data but also the sound collected together with the image data A and the image data B may be output.

  When the display image data is a still image, when the instructor or the like presses the button 504, the display control unit 32 displays the next display image data in the image data column 502. Similarly, the display control unit 32 can display the previous display image data in the image data field 502 by a button operation. The display control unit 32 may display the display image data in the image data column 502 one after another at a predetermined time interval.

  Next, the measurement data column 503 will be described. In FIG. 18, the respective accelerations in the x direction, the y direction, and the z direction are displayed in time series. In FIG. 18, acceleration in any one or more directions may be displayed by an operation of a leader or the like among the x direction, the y direction, or the z direction. The measurement data is continuously displayed. For example, when the display image data is a still image, the measurement data is measured by the acceleration measuring device 50 at the timing when the omnidirectional image capturing device 60 captures the image. Acceleration may be displayed.

  The display control unit 32 scales and displays the measurement data in the measurement data column 503 so that all the measurement values included in the measurement data are displayed in the measurement data column 503. Note that the display control unit 32 may display measurement values for a certain amount of time in the measurement data column 503 and switch them at predetermined intervals.

  As illustrated in FIG. 18A, the display control unit 32 causes the measurement data column 503 to display a bar 505 indicating the measurement value associated with the display frame data of the frame image displayed in the image data column 502. . For example, when the measurement values for the whole or a certain time are displayed in the measurement data column 503 and the display image data is reproduced, the display control unit 32 is associated with the display frame data of the frame image being displayed. A bar 505 is superimposed on the measured value.

  Therefore, the display control unit 32 moves the bar 505 to the right in the measurement data column 503 as time elapses or as the reproduction of the moving image proceeds. When a leader or the like moves the bar 505, the operation input accepting unit 33 accepts the operation, and the display control unit 32 displays a frame image of the display frame data associated with the measurement value indicated by the moved bar 505. May be displayed in the image data field 502.

  Further, as shown in FIG. 18B, the measurement value associated with the display frame data of the frame image displayed in the image data column 502 and the measurement values before and after the measurement value are displayed in the measurement data column 503. The In the measurement data column 503, for example, only measured values at intervals of about 1 second to several seconds are displayed simultaneously. The broken line measurement data indicates that the measurement data is continuous and is not displayed on the screen.

  In the case of FIG. 18B, the display control unit 32 displays the measurement value associated with the display frame data of the frame image displayed in the image data field 502 as the measurement data according to the progress of the moving image or the still image. Displayed in the center of the column 503. Since the display image data and the measurement data are associated with each other by the server 10, the display control unit 32 measures at least the measurement value associated with the display frame data of the frame image displayed in the image data field 502. It can be displayed in the data column 503. In addition, when the moving image or the still image in the image data field 502 is stopped, the display control unit 32 also stops displaying the measurement data.

≪About evaluation of measurement data≫
Here, when the measurement data is acceleration, a peak appears periodically in a general way of walking, and the height of the peak does not vary much. On the other hand, if the walking function deteriorates due to age, injury, or the like, the peak of acceleration does not appear periodically or the height of the peak tends to fluctuate.

  For this reason, the instructor or the like may be able to evaluate the walking motion only by looking at the display image data, but can further support the measurement data that the walking motion is not appropriate. In addition, a leader or the like can identify a place where the measurement data is disturbed and can check the display image data. Although the display image data is displayed only for a frame image at a certain moment, since the measurement data has a list property, an instructor or the like can quickly display a frame image on which an inappropriate walking motion is displayed based on the measurement data. You can search by

  In the present embodiment, the acceleration is described as an example. However, the angular velocity detected by the gyro sensor 206, the position detected by the GPS receiver 213, the azimuth detected by the acceleration / direction sensor 205, etc. It is also possible to evaluate whether the walking motion is appropriate.

  For example, the angular velocity detected by the gyro sensor 206 indicates the speed of change of the yaw angle, pitch angle, and roll angle when the pedestrian 200 walks. In this case as well, as with acceleration, it is known that peaks appear periodically in a general way of walking and the height of the peak does not vary much.

  Further, since the acceleration is obtained by differentiating the position twice, the position detected by the GPS receiver 213 and the like can be used as information equivalent to the acceleration. In the case of turning walking, the azimuth constantly changes, but acceleration of the change in azimuth can be obtained by differentiating the azimuth twice. In this case as well, the peak period and height are a guide for evaluation. In addition, in these measurement data, even if the peak is periodic and the height fluctuation is small, if the peak is extremely large, there may be an improvement in walking motion. Can confirm the display image data.

  The display control unit 32 of the display device 30 may calculate and display the area between the acceleration and the axis of acceleration = 0. For example, when the pedestrian 200 walking in the shape of figure 8 is turning, the y direction is the acceleration in the left-right direction of the pedestrian 200, so that the acceleration is an area on the positive side and an area on the negative side. Indicates left and right blurring with respect to the line L. If the pedestrian 200 walks on the line L, the area on the positive side and the area on the negative side are almost equal. Therefore, when the area on the positive side is significantly different from the area on the negative side, it can be seen that the balance between the left leg and the right leg of the pedestrian 200 is poor.

  In this way, a leader or the like can obtain a lot of information for evaluating the walking motion of the pedestrian 200 by looking at the measurement data and the display image data whose time axes are synchronized.

<Rotation of spherical image>
Since the 360 ° image around the omnidirectional image capturing device 60A or 60B is reflected in the display image data, the image is distorted when the entire image is displayed, and the walking motion is difficult to understand. For this reason, it is preferable that the display control unit 32 displays only the range in which the pedestrian 200 is shown in the image data field 502. Such processing is expressed as rotation of the omnidirectional image, trimming, extraction or extraction of the pedestrian 200, or the like. The rotation of the omnidirectional image refers to determining the range displayed in the image data field 502 by rotating the omnidirectional image in the horizontal direction (in the longitude direction).

  FIG. 19 is a flowchart illustrating an example of processing in which the image rotation unit 34 specifies a range in which the pedestrian 200 is shown and displays the range in the image data field 502. The processing in FIG. 19 is started, for example, when the display device 30 receives display image data and the display image data is reproduced.

  First, the image rotation unit 34 specifies the position of the foot of the pedestrian 200 (S901). In addition, although a leg means the parts of the shoes etc. which the pedestrian 200 is wearing, when not wearing shoes etc., it is a part from a heel to a toe. The image rotation unit 34 may specify a leg below the waist or below the knee as a foot.

  Here, when the display image data is a moving image, the image rotating unit 34 can detect the pedestrian 200 by detecting a moving object if the pedestrian 200 performs an operation that steps several times. Further, the image rotation unit 34 may detect the pedestrian 200 even if the face is recognized. Although there is a possibility that a person other than the pedestrian 200 exists around the omnidirectional image capturing device 60, the image rotation unit 34 may detect the moving object occupying the largest pixel range as the pedestrian 200. In addition, the image rotation unit 34 may learn a foot image by machine learning, for example, and recognize the foot from the display image data.

  When the pedestrian 200 starts walking from a predetermined position, the initial position of the pedestrian 200 is a predetermined position in the horizontal direction of the display image data. Therefore, in this case, the image rotation unit 34 can easily identify the pedestrian 200 to be measured.

  On the other hand, in the case of a still image, the image rotation unit 34 can recognize the face and detect the pedestrian 200. Moreover, the image rotation part 34 may detect the pedestrian 200 from the display image data of each still image by pattern matching etc., for example, a leader etc. may input the position of the pedestrian 200.

  Next, the image rotation unit 34 calculates the angular velocity of the pedestrian 200 per unit time from the change in the position of the foot due to the walking of the pedestrian 200 (S902).

  Here, the angular velocity will be described with reference to FIG. FIG. 20 is a diagram schematically illustrating an example of the position of the foot 511 of the pedestrian 200 in the top view of the omnidirectional image.

In FIG. 20, as an example, the north direction is the reference. At this time, the foot of the pedestrian 200 can take positions (angles in which the north direction is zero degrees) θ ₁ and θ ₂ in the range of 360 ° in the longitude direction. Therefore, an average of the _two feet positions θ ₁ and θ ₂ of the pedestrian 200 in one frame image is set as the position θ of the foot 511.

  Then, for example, the average amount of change in the position θ in the past several seconds is calculated every predetermined time (for example, every time the frame is updated). Thereby, the image rotation part 34 can calculate the angular velocity of the pedestrian 200 with little fluctuation | variation. The angular velocity can be referred to as walking speed (traveling speed information) viewed from the omnidirectional image capturing device 60. The walking speed in the x direction in the real space can be obtained by integrating the acceleration in the x direction.

  Next, the image rotation unit 34 rotates the frame image at the angular velocity calculated in step S902 (S903). That is, when the frame data included in the display image data is switched, the image rotation unit 34 rotates the display range (predetermined range) of the image data column 502 around the position where the pedestrian 200 has moved at the angular velocity. Trim the frame image. Thereby, it is possible to make the pedestrian 200 appear in substantially the center of the frame image trimmed by the image rotation unit 34.

  The display control unit 32 displays the frame image of the display range trimmed by the image rotation unit 34 in the image data field 502 (step S904). For this reason, a leader etc. can evaluate walking action of the pedestrian 200, without searching for the pedestrian 200 from the omnidirectional image by which the range of 360 degrees was imaged.

  In addition, since a camera with a normal angle of view is picked up so that the pedestrian 200 is reflected, processing such as rotation and trimming is not necessary. Further, the server 10 or the omnidirectional image capturing device 60 may have the function of the image rotation unit 34. In this case, the communication load and communication time when display image data, image data A, and image data B are transmitted from the server 10 or the omnidirectional image capturing device 60 to the display device 30 can be reduced.

<Another example of rotation of a spherical image>
Here, the case where the pedestrian 200 is recognized by recognizing the face from the frame image and the frame image is rotated will be described. FIG. 21 is a flowchart illustrating another example of processing for displaying the range in which the pedestrian 200 is shown in the image data field 502. The process of FIG. 21 is started, for example, when the display device 30 receives display image data and the display image data is reproduced.

  First, the image rotation unit 34 recognizes a face for each omnidirectional image (that is, a frame image) (step S1001). FIG. 22 is an example of a diagram for schematically explaining the recognition of the face 512. The recognition of the face 512 is performed by setting an appropriate feature amount and learning by a learning identification device.

  For example, the image rotation unit 34 can use Haar-like feature values, LBP (Local Binary Patterns) feature values, HOG (Histogram of Oriented Gradients) features, and the like as feature values. The image rotation unit 34 can use SVM (Support Vector Machines), AdaBoost, or the like as a learning method of the learning identification device. However, the present invention is not limited to these, and the image rotation unit 34 can use various feature quantities that can recognize the face 512 and learning methods of the learning identification device.

  Next, the image rotation unit 34 trims the image in the display range 513 (predetermined range) in the image data field 502 around the face 512 from the omnidirectional image (frame image) (step S1002). Thereby, it is possible to make the pedestrian 200 appear in the center of the image taken out by the image rotation unit 34.

  The display control unit 32 displays the frame image of the display range 513 trimmed by the image rotation unit 34 in the image data field 502 (step S1003). For this reason, a leader etc. can evaluate walking action of the pedestrian 200, without searching for the pedestrian 200 from the omnidirectional image by which the range of 360 degrees was imaged.

  Note that the image rotation unit 34 does not determine and crop the position of the face 512 for each omnidirectional image, but calculates the moving speed in the longitude direction from the position of the face 512 of several past omnidirectional images. Thus, the display range may be determined from the omnidirectional image to be displayed next. Thereby, the position of the face 512 is less likely to fluctuate, so that a smooth moving image can be displayed.

  As described above, in the image display system 100 according to the present embodiment, since the pedestrian 200 turning around the plurality of omnidirectional image capturing devices 60 and turning left and right is imaged, various walking motions of the pedestrian 200 are performed. Can be imaged. In the image display system 100 according to the present embodiment, the captured image of the pedestrian 200, an image facing the front, and the like are displayed on the display device 30 together with measurement data such as acceleration of the pedestrian 200. To do.

  For this reason, according to the image display system 100 which concerns on this embodiment, an instructor etc. can display the image which is easy to evaluate the walking motion of the pedestrian 200. FIG.

<Other System Configurations of Image Display System 100>
Here, another system configuration of the image display system 100 will be described. FIG. 23 is a diagram illustrating another example of the system configuration of the image display system 100.

  FIG. 23A shows another example of the system configuration of the image display system 100. In the image display system 100 illustrated in FIG. 23A, the server 10 includes a display 108. In the configuration shown in FIG. 23A, the server 10 has various functions of the display device 30. That is, the server 10 displays the measurement data and the display image data by synchronizing the time axis of the measurement data and the image data and creating the display image data, and rotating the frame image.

  FIG. 23B is another example of the system configuration of the image display system 100. The image display system 100 illustrated in FIG. 23B does not include the server 10 and the display device 30. In the configuration shown in FIG. 23B, the acceleration measuring device 50 has various functions of the server 10 and the display device 30.

  Here, the overall operation in the image display system 100 shown in FIG. 23B will be described. FIG. 24 is a sequence diagram illustrating an example of the overall operation of the image display system 100 when the acceleration measuring device 50 displays measurement data and display image data.

  First, when starting measurement of walking motion, a leader or a pedestrian 200 inputs an operation for starting measurement of walking motion to the acceleration measuring device 50. Then, the acceleration measuring device 50 receives an operation of a leader or a pedestrian 200 (step S1101).

  Next, the communication unit 51 of the acceleration measuring device 50 transmits an imaging start request to the omnidirectional image capturing devices 60A and 60B (steps S1102 and S1103).

  Next, the acceleration detector 52 of the acceleration measuring device 50 starts measuring acceleration (step S1104). In addition, the imaging units 62 of the omnidirectional imaging devices 60A and 60B start imaging the pedestrian 200 (steps S1105 and S1106), respectively.

  Next, when the walking of the pedestrian 200 is finished, the leader or the pedestrian 200 inputs an operation for ending the measurement of the walking motion to the acceleration measuring device 50. Then, the acceleration measuring device 50 receives the operation of the instructor or the pedestrian 200 (step S1107).

  Next, the communication unit 51 of the acceleration measuring device 50 transmits an imaging end request to the omnidirectional image capturing devices 60A and 60B (steps S1108 and S1109).

  When the imaging unit 62 of the omnidirectional image imaging devices 60A and 60B receives the imaging end request from the communication unit 61, the imaging unit 62 ends imaging. Then, the communication units 61 of the omnidirectional image capturing devices 60A and 60B transmit the image data A and the image data B to the acceleration measuring device 50, respectively (steps S1110 and S1111).

  The acceleration data processing unit 13 of the acceleration measuring device 50 processes the measurement data in the same manner as in step S118 of FIG. 8 (step S1112).

  Next, the image adjustment unit 12 of the acceleration measuring device 50 synchronizes the time axes of the measurement data and the image data, similarly to step S119 of FIG. 8 (step S1113).

  Next, the display image creation unit 16 of the acceleration measuring device 50 creates display image data, similarly to step S120 in FIG. 8 (step S1114).

  Next, the acceleration measuring device 50 displays the measurement data and the display image data on the display 214 of the own device (step S1115). The acceleration measuring apparatus 50 may transmit the measurement data and the display image data to a terminal such as a leader (server 10 or the like) (step S1116). In this case, a terminal such as a leader displays measurement data and display image data (step S1117).

  Therefore, according to the configuration shown in FIG. 23B, the system configuration of the image display system 100 can be simplified. The pedestrian 200 walks while wearing the acceleration measuring device 50. After the end of walking, the pedestrian 200 can receive guidance from the instructor by passing the acceleration measuring device 50 to the instructor or the like.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, in the present embodiment, the measurement data and the display image data are displayed after the measurement data is measured and the image data is captured. However, the measurement data and the display image data may be displayed in real time. In this case, the displayed measurement data and display image data are taken at substantially the same time.

  In addition, the configuration example of FIG. 6 and the like shown in the above embodiment is mainly used for easy understanding of the processing of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50. It is divided according to the function. For example, a plurality of servers 10 may exist, and a plurality of servers 10 may cooperate to perform the processing of this embodiment. Further, the storage unit 1000 of the server 10 does not need to be included in the server 10, and the storage unit 1000 only needs to be in a place where the server 10 can read and write data in the storage unit 1000.

  In FIG. 6, the functions of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50 have been described by being divided into several processing units. However, the present invention is not limited by the way of dividing or the name of each processing unit. The processing of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50 can be divided into more processing units depending on the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes.

DESCRIPTION OF SYMBOLS 10 Server 12 Image adjustment part 16 Display image creation part 30 Display apparatus 32 Display control part 33 Operation input reception part 34 Image rotation part 50 Acceleration measuring apparatus 60 Omnispherical image imaging device 100 Image display system

JP 2010-172394 A

Claims (17)

An image display system having an information processing device, a plurality of wide-angle imaging devices capable of imaging each of the surroundings in the horizontal direction, and a measuring device that measures the movement of a moving object,
Acquisition means for acquiring a plurality of image data generated by imaging the target by the plurality of wide-angle imaging devices and measurement data generated by measuring the operation of the target by the measuring device;
Association means for associating each of the plurality of image data with the measurement data;
Determining means for determining image data to be displayed among the plurality of image data respectively associated with the measurement data;
Display means for displaying the determined image data and the measured data on a display device;
An image display system. The determining means includes
2. The image display system according to claim 1, wherein, for each of the plurality of image data, an image area including the target is specified from the image data, and image data having the largest specified image area is determined as a display target. . The determining means includes
The image display system according to claim 1, wherein among the plurality of image data, image data in which the target is imaged from a predetermined direction is determined as a display target. The determining means includes
For one image data of the plurality of image data, the image area including the object is specified from the frame image data included in the one image data, and the size of the image area in the preceding and following frame image data is changed. The image display system according to claim 3, wherein whether or not the one image data is to be displayed is determined. The plurality of wide-angle imaging devices include:
Imaging each of the objects traveling around the plurality of wide-angle imaging devices while changing the direction or changing the direction;
The association means includes
5. The image display system according to claim 1, wherein the measurement data is associated with each of the plurality of pieces of image data obtained by imaging the object that progresses while changing the direction or changing the direction. 6. The information processing apparatus includes:
Requesting the measurement device to start measurement and end measurement, and having an instruction means to request imaging start and imaging end of each of the plurality of wide-angle imaging devices,
The association means includes
Based on the imaging start of each of the plurality of image data and the measurement start of the measurement data, and the imaging end of each of the plurality of image data and the measurement end of the measurement data, each of the plurality of image data, and the measurement data And the image display system according to any one of claims 1 to 5. The display device
Receiving means for receiving at least one of measurement start of the target motion or imaging start of the target;
Notification means for notifying the information processing apparatus of at least one of the measurement start or the imaging start received by the reception means;
The image display system according to claim 6, comprising: The measurement data includes a plurality of measurement values obtained by measuring the operation of the object,
The measuring device is
When the measurement value equal to or greater than a predetermined threshold is detected, the measurement data is started and the measurement means for requesting each of the plurality of wide-angle imaging devices to start imaging,
The wide-angle imaging device
Upon receiving a request to start imaging by the measuring unit, the imaging unit starts imaging the target and starts generating the image data.
The image display system according to claim 1, comprising: The measuring means includes
When the measurement value less than a predetermined threshold is detected, the generation of the measurement data is terminated, and the end of imaging is requested to each of the plurality of wide-angle imaging devices,
The imaging means includes
The image display system according to claim 8, wherein upon receiving a request to end imaging by the measurement unit, imaging of the target is ended and generation of the image data is ended. The wide-angle imaging device
The image display system according to any one of claims 1 to 9, further comprising: a first notification unit that notifies the start of imaging of the target. The measuring device is
11. The image display system according to claim 1, further comprising a second notification unit that notifies that measurement of the movement of the target is started. An information processing device capable of communicating with a plurality of wide-angle imaging devices capable of imaging the surroundings in the horizontal direction and a measuring device that measures the movement of a moving object,
Acquisition means for acquiring a plurality of image data generated by each of the plurality of wide-angle imaging devices imaging the target, and measurement data generated by the measurement device measuring an operation of the target;
Association means for associating each of the plurality of image data with the measurement data;
Determining means for determining image data to be displayed among the plurality of image data respectively associated with the measurement data;
Display means for displaying the determined image data and the measured data on a display device;
An information processing apparatus. An information processing device capable of communicating with a plurality of wide-angle imaging devices capable of imaging the surroundings in the horizontal direction,
Measuring means for measuring the movement of the moving object and generating measurement data;
Acquisition means for acquiring a plurality of image data generated by each of the plurality of wide-angle imaging devices imaging the target;
Association means for associating each of the plurality of image data with the measurement data;
Determining means for determining image data to be displayed among the plurality of image data respectively associated with the measurement data;
Display means for displaying the determined image data and the measured data on a display device;
An information processing apparatus. In an imaging device capable of imaging the surroundings in the horizontal direction and capable of communicating with another imaging device, an information processing device, and a measuring device that measures the movement of a moving object,
Imaging means for generating first image data by imaging the object;
Acquisition means for acquiring second image data generated by imaging the target by the other imaging device and measurement data generated by measuring the operation of the target by the measurement device;
Transmitting means for transmitting the first image data, the second image data, and the measurement data to the information processing apparatus;
An imaging apparatus having An information processing device capable of communicating with a plurality of wide-angle imaging devices capable of imaging the surroundings in the horizontal direction and a measuring device that measures the movement of a moving object,
Acquisition means for acquiring a plurality of image data generated by imaging the target by the plurality of wide-angle imaging devices, and measurement data generated by measuring the operation of the target by the measuring device;
Association means for associating each of the plurality of image data with the measurement data;
Determining means for determining image data to be displayed among the plurality of image data respectively associated with the measurement data;
Display means for displaying the determined image data and measurement data on a display device;
Program to function as. An information processing device capable of communicating with a plurality of wide-angle imaging devices capable of imaging the surroundings in the horizontal direction,
Measuring means for measuring the movement of the moving object and generating measurement data,
Acquisition means for acquiring a plurality of image data generated by each of the plurality of wide-angle imaging devices imaging the target;
Association means for associating each of the plurality of image data with the measurement data;
Determining means for determining image data to be displayed among the plurality of image data respectively associated with the measurement data;
Display means for displaying the determined image data and measurement data on a display device;
Program to function as. An imaging device capable of imaging the surroundings in the horizontal direction, the imaging device capable of communicating with another imaging device, an information processing device, and a measuring device that measures the movement of a moving object,
Imaging means for generating first image data by imaging the object;
Acquisition means for acquiring second image data generated by imaging the target by the other imaging device and measurement data generated by measuring the operation of the target by the measurement device;
Transmitting means for transmitting the first image data, the second image data, and the measurement data to the information processing apparatus;
Program to function as.