WO2018098867A1

WO2018098867A1 - Photographing apparatus and image processing method therefor, and virtual reality device

Info

Publication number: WO2018098867A1
Application number: PCT/CN2016/111544
Authority: WO
Inventors: 刘鑫
Original assignee: 歌尔科技有限公司
Priority date: 2016-11-29
Filing date: 2016-12-22
Publication date: 2018-06-07
Also published as: CN106507092A

Abstract

A photographing apparatus and an image processing method therefor, and a virtual reality device. The photographing apparatus comprises a processor unit, a communication module, and at least one group of binocular cameras, wherein the processor unit comprises at least one processor, the at least one group of binocular cameras transmits collected image data to the processor unit for processing, and the processing unit transmits a corresponding processing result to the communication module for sending. The virtual reality device comprises the photographing apparatus and a host computer, wherein a processor unit of the photographing device transmits processing results obtained by means of processing various types of data to the host computer via a communication module.

Description

Camera device and image processing method thereof, virtual reality device

Technical field

The present invention relates to the field of image acquisition technology, and more particularly to an image pickup apparatus, an image processing method of the image pickup apparatus, and a virtual reality apparatus having the same.

Background technique

Virtual reality technology is abbreviated as VR technology, which uses computer simulation to generate a virtual world in a three-dimensional space, providing users with simulations of visual, auditory, tactile and other senses, so that users can be as immersive as they can, and can observe three times in a timely and unrestricted manner. Transactions within the space.

The spatial positioning system in virtual reality mainly recognizes the position and posture of the participants by tracking the spatial position of peripherals such as helmets and handles.

In the existing spatial positioning system, the infrared signals emitted by the infrared emitters installed on these peripherals are mainly collected by the camera device for positioning and tracking of peripherals.

Summary of the invention

According to a first aspect of the present invention, there is provided an image pickup apparatus comprising a processor unit, a communication module, and at least one set of binocular cameras, the processor unit including at least one processor, the at least one set of binoculars The camera transmits the collected image data to the processor unit for processing, and the processor unit transmits the corresponding processing result to the communication module for transmission.

According to a second aspect of the present invention, a virtual reality device is provided, comprising: a host device and a camera device according to the first aspect of the present invention, wherein the host establishes a communication connection with a communication module of the camera device, The processor unit of the camera device transmits the processing result obtained by processing various types of data to the host through the communication connection.

According to a third aspect of the present invention, there is provided an image processing method of an image pickup apparatus according to the first aspect of the present invention, wherein the processor unit:

Receiving image data collected by at least one set of binocular cameras;

Preprocessing the image data;

Generating a depth image based on the preprocessed image data;

Obtaining position information of the positioned object based on the depth image; and

The location information is transmitted to the communication module of the imaging device as a processing result corresponding to the image data for transmission.

Other features and advantages of the present invention will become apparent from the Detailed Description of the <RTIgt;

DRAWINGS

The accompanying drawings, which are incorporated in FIG

1 is a block schematic diagram of an embodiment of an image pickup apparatus according to the present invention;

2 is a block schematic diagram of an embodiment of a virtual reality device in accordance with the present invention;

3 is a flow chart showing an embodiment of an image processing method of an image pickup apparatus according to the present invention.

Description of the reference signs:

U101-application processor; U102-coprocessor;

U3-storage unit; U4-inertial measurement unit;

U5-communication module; U6-power management chip;

U7-audio codec chip; B1-battery;

J1-USB socket; S1-speaker module;

M1-microphone module; C1-first group of binocular cameras;

C2-second group binocular camera; U1-processor unit.

detailed description

Various exemplary embodiments of the present invention will now be described in detail with reference to the drawings. It should be noted that the relative arrangement of the components and steps, numerical expressions and numerical values set forth in the embodiments are not intended to limit the scope of the invention unless otherwise specified.

The following description of the at least one exemplary embodiment is merely illustrative and is in no way

Techniques, methods and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but the techniques, methods and apparatus should be considered as part of the specification, where appropriate.

In all of the examples shown and discussed herein, any specific values are to be construed as illustrative only and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

It should be noted that similar reference numerals and letters indicate similar items in the following figures, and therefore, once an item is defined in one figure, it is not required to be further discussed in the subsequent figures.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a block schematic diagram of an embodiment of an image pickup apparatus according to the present invention.

According to FIG. 1, the camera device includes a processor unit U1, a communication module U5, and two sets of binocular cameras C1, C2.

The image data collected by each group of binocular cameras C1 and C2 is transmitted to the processor unit U1 for processing.

The processor unit U1 can configure each group of binocular cameras C1, C2, for example, via an I2C bus.

The processor unit U1 transmits the processing result of various processing, including the processing result obtained by processing the image data, to the communication module U5 for transmission. In this way, after the imaging device establishes a communication connection with the host through the communication module U5, the processing result obtained by the processor unit U1 can be sent to the host for use by the host, thereby reducing the processing load of the host.

In order to improve the convenience of use, in this embodiment, the communication module U5 is a wireless communication module, so that various types of processed data can be transmitted by establishing a wireless communication connection with the host.

In another embodiment, the communication module U5 may also be a wired communication module, such as a USB communication module or the like.

Each set of binocular cameras includes two cameras of the same type, which are equivalent to the left and right eyes for information capture in three dimensions.

In the embodiment shown in FIG. 1, the first set of binocular cameras C1 is a standard camera, for example The sensor model is the OV9281 camera for gesture capture; the second group of binocular camera C2 uses a fisheye lens, such as the sensor model OV7251, for depth of field.

Taking the application scenario as a user to experience the VR game with the wireless controller with infrared light as an example, the two sets of binocular cameras C1 and C2 will capture the infrared light emitted by the handle infrared light at the set frequency, and the first group of binocular cameras C1 will gesture. Capture, the second group of binocular cameras C2 obtains the depth of field, and together to complete the positioning of the handle action. The processor unit U1 processes the image data captured by the two sets of binocular cameras C1 and C2, and further obtains the position information of the handle in the space, and sends the processed position information to the host of the virtual reality device through the communication module U5. After the host gets the location information of the handle, it can map it to the VR scene to implement the corresponding VR operation.

In this embodiment, the camera device may further include an Inertial Measurement Unit (IMU) U4, and the inertial measurement unit U4 also transmits the collected motion data to the processor unit U1 for processing, and the processor unit U1 may The corresponding processing result is also transmitted to the communication module U5 for transmission.

The inertial measurement unit (IMU) U4 is a device for measuring the three-axis attitude angle (or angular rate) of the object and the acceleration. The gyroscope, the accelerometer, and the geomagnetic sensor are the main components of the IMU. An IMU may, for example, comprise three single-axis accelerometers and three single-axis gyroscopes, the accelerometer detecting an acceleration signal of an independent three-axis of the object in the carrier coordinate system, and the gyroscope detecting the angular velocity signal of the carrier relative to the navigation coordinate system, The angular velocity and acceleration of the object in three-dimensional space are measured, and the posture of the object is calculated.

The processor unit U1 includes at least one processor for processing data collected by each group of binocular cameras C1, C2 and inertial measurement unit U4.

In this embodiment, the processor unit U1 includes an application processor U101 and a coprocessor U102 to perform certain tasks through the coprocessor U102, such as monitoring each group of binocular cameras C1, C2, and an inertial measurement unit. U4, etc., thereby reducing the burden on the application processor U101.

The coprocessor U102 is communicatively coupled to the application processor U101, for example, via a USB 3.0 bus, and the communication module U5 is communicatively coupled to the application processor U101, for example, via an I2C bus. In this way, the coprocessor U102 can transmit the data preprocessed by the coprocessor U102 to the application processor U101 for further processing, and the application processor U101 transmits the processed processing result to the communication module U5. send.

Therefore, in this embodiment, at least one set of binocular cameras C1, C2 can be connected to the coprocessor U102 through the MIPI interface, for example, to preprocess the image data collected by the binocular cameras C1, C2 by the coprocessor U102. .

In embodiments where there is no MIPI interface at either end, the connections at both ends can be achieved through the MIPI bridge chip.

Also, in this embodiment, the inertial measurement unit U4 can be connected to the coprocessor U102 via the SPI interface, for example, to preprocess the motion data collected by the inertial measurement unit U4 by the coprocessor U102.

The camera device may further include an audio codec chip U7, a microphone module M1, and a speaker module S1. The microphone module M1 is connected to the processor unit U1 through an audio coding channel of the audio codec chip U7, and the processor unit U1 passes the audio. The audio decoding channel of the codec chip U7 is connected to the speaker module S1. Thus, voice communication can be performed by the image pickup apparatus of the present invention.

In an embodiment in which the processor unit U1 has an application processor U101 and a coprocessor U102, the audio codec chip U7 can be connected, for example, to the application processor U101 of the processor unit U1 via an I2S bus.

The camera device can also be powered by a battery to improve the reliability and convenience of the camera.

Therefore, in this embodiment, the camera device may further include a battery B1, a power management chip (PMIC/PMU) U6, and a USB socket J1. The battery B1 supplies power to each of the power devices of the camera device via the battery management chip U6. J1 is connected to power management chip U6 and processor unit U1 for data transmission, and power management chip U6 is communicatively coupled to processor unit U1 for transmitting control commands and/or status information.

The working mode between the power management chip U6 and the processor unit U1 is as follows:

After detecting that the VBUS pin of the USB socket J1 is powered on, the power management chip U6 performs an interaction with the external device inserted into the USB socket J1, and after the interaction, if it is judged that the external device is a charger, the charging channel is turned on. The charger charges the battery; if it is judged that the inserted external device is a USB host, the processor unit U1 is notified through a communication channel with the processor unit U1 to establish a USB connection between the processor unit U1 and the external device. .

In an embodiment where processor unit U1 includes application processor U101 and coprocessor U102, the power management chip U6 can be communicatively coupled to application processor U101, for example, via an I2C bus. The USB socket J1 can be connected only to the application processor U101. The USB socket J1 can also be connected to the application processor U101 and the coprocessor U102 at the same time, where the connection selection can be made by communication between the application processor U101 and the coprocessor U102.

The camera device may further include an isolation circuit for connecting the power management chip U6 to the USB socket J1 through the isolation circuit, thereby connecting the circuit between the power management chip U6 and the USB socket J1 and between the processor unit U1 and the USB socket J1. The high speed communication circuit is isolated.

In this embodiment, the image capturing apparatus may further include a storage unit U3, and the storage unit U3 includes at least one memory for expanding the storage space of the processor unit U1.

The storage unit U3 includes, for example, at least one of double rate synchronous dynamic random access memory (DDR) and FLASH.

In embodiments where processor unit U1 includes application processor U101 and coprocessor U102, the memory unit U3 can be directly coupled to the read/write pins of coprocessor U102.

2 is a block schematic diagram of one embodiment of a virtual reality device in accordance with the present invention.

According to FIG. 2, the virtual reality device comprises a host 210 and an imaging device according to the invention, which is labeled 220 in this embodiment.

The imaging device 220 establishes a communication connection with the host 210 through its communication module U5, and further transmits the processing result obtained by processing the various types of data by the processor unit U1 of the imaging device 220 to the host 210 for use by the host 210.

For the virtual reality device of the present invention, since the imaging device 220 itself undertakes the main calculation task, the burden on the host 210 can be greatly reduced, thereby effectively solving the problem of heat generation of the host. Therefore, the virtual reality device of the present invention can adopt The design of the host 210 on the wearing portion of the virtual reality device does not cause user discomfort due to severe heat generation.

In other embodiments, the host 210 can also be disposed in the mobile handle in communication with the camera 220 and the headset.

In other embodiments, the host 210 can also be a fixed PC in communication with the camera 220 and the headset.

FIG. 3 shows a kind of image processing performed by the processor unit U1 of the image pickup apparatus according to the present invention. A schematic flow chart of the implementation method.

According to FIG. 3, the image processing method may include the following steps:

Step S301, receiving image data collected by at least one set of binocular cameras C1, C2.

The processor unit U1 receives image data acquired by at least one set of binocular cameras C1, C2, for example via an MIPI bus. Step S302, preprocessing the received image data to improve image quality.

The pre-processing may include at least one of grayscale processing, enhancement processing, filtering processing, binarization processing, white balance processing, demosaic processing, gamma correction processing, and the like.

This pre-processing can be performed, for example, by an image acquisition engine (IAE) integrated by the processor unit.

Step S303, generating a depth image based on the preprocessed image data.

Each pixel value in the depth image is used to represent the distance of a point in the scene relative to the camera.

This can produce depth images, for example, based on pre-processed image data, through intelligent parallax mapping and refinement algorithms, and the like.

Step S304, based on the depth image, obtain position information of the positioned object.

The step S304 can be performed, for example, by a computer Vision Engine (CVE) integrated by the processor unit, and the computer vision engine processes the depth image through the DSP and the computer vision algorithm to obtain position information of the positioned object. The object to be positioned is, for example, a helmet, a handle, or the like that is marked with infrared light.

In step S305, the location information is transmitted as a processing result of the corresponding image data to the communication module U5 of the imaging device for transmission.

Through this step, the camera device can send the processing result obtained by the processor unit U1 processing the received image data to the host for use by the host, so that the host does not need to occupy resources to collect image data collected by at least one group of binocular cameras. Processing, thereby reducing the burden on the host.

In the embodiment shown in FIG. 1, the processor unit U1 includes an application processor U101 and a coprocessor U102. The above steps S301 to S304 may all be completed by the coprocessor U102, and the application processor U101 is only responsible for processing the result. Integrating and transmitting the processing result through the communication module U5; or the coprocessor U102 may perform the above steps S301 and S302, or The above steps S301 to S303 are performed, and the remaining steps are performed by the application processor U101.

In addition, the processing of the motion data collected by the inertial measurement unit (IMU) by the processor unit U1 may be, for example, transmission to the integrated processor unit through fast interrupt response (FIQ), further integrated into the processor unit. In the processor, a General Purpose Processor (GPP) performs processing to generate a quaternion, and transmits the quaternion as a processing result of the corresponding motion data to the communication module U5 for transmission.

The various embodiments in the present specification are described in a progressive manner, and the same or similar parts between the various embodiments may be referred to each other. Each embodiment focuses on differences from other embodiments, and each implementation The examples can be used alone or in combination with each other as needed.

While the invention has been described in detail with reference to the preferred embodiments of the present invention, it is understood that It will be appreciated by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

A camera device, comprising: a processor unit (U1), a communication module (U5), and at least one set of binocular cameras (C1, C2), the processor unit (U1) comprising at least one processor ( U101, U102), the at least one set of binocular cameras (C1, C2) transmit the collected image data to the processor unit (U1) for processing, and the processor unit (U1) will process the corresponding processing result Transfer to the communication module (U5) for transmission.
The image pickup apparatus according to claim 1, wherein the image pickup apparatus includes at least two sets of binocular cameras (C1, C2), and wherein one set of binocular cameras (C2) employs a fisheye lens.
The image pickup apparatus according to claim 1 or 2, wherein said processor unit (U1) comprises an application processor (U101) and a coprocessor (U102), said at least one set of binocular cameras (C1, C2) Connected to the coprocessor (U102), the coprocessor (U102) is communicatively coupled to the application processor (U101), and the communication module (U5) is communicatively coupled to the application processor (U101).
The image pickup apparatus according to claim 3, wherein said coprocessor (U102) and said application processor (U101) are communicably connected by a USB bus.
The image pickup apparatus according to any one of claims 1 to 4, wherein the image pickup apparatus further comprises an inertial measurement unit (U4) that transmits the collected motion data to the camera The processor unit (U1) performs processing, and the processor unit (U1) transmits the corresponding processing result to the communication module (U5) for transmission.
The image pickup apparatus according to any one of claims 1 to 5, further comprising an audio codec chip (U7), a microphone module (M1), and a speaker module (S1), a microphone module (M1) is connected to the processor unit (U1) through an audio encoding channel of the audio codec chip (U7), and the processor unit (U1) passes through the audio codec chip (U7) An audio decoding channel is coupled to the speaker module (S1).
The image pickup apparatus according to any one of claims 1 to 6, wherein the communication module (U5) is a wireless communication module.
The image pickup apparatus according to any one of claims 1 to 7, characterized in that the image pickup apparatus further comprises a battery (B1), a power management chip (U6), and a USB socket (J1), the battery (B1) Powering the power device of the camera device via the battery management chip (U6), the USB socket (J1) being respectively connected to the power management chip (U6) and the processor unit (U1), A power management chip (U6) is communicatively coupled to the processor unit (U1).
A virtual reality device, comprising: a host (210) and the camera device (220) according to any one of claims 1 to 8, the host (210) communicating with the camera device (220) The module (U5) establishes a communication connection, and the processor unit (U1) of the camera device (220) transmits the processing result obtained by processing various types of data to the host (210) through the communication connection.
The image processing method of the image pickup apparatus according to any one of claims 1 to 8, wherein the processor unit:

Receiving image data collected by at least one set of binocular cameras (C1, C2);

Preprocessing the image data;

Generating a depth image based on the preprocessed image data;

Obtaining position information of the positioned object based on the depth image; and

The location information is transmitted to the communication module of the imaging device as a processing result corresponding to the image data for transmission.