Detailed Description
Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
< hardware configuration >
As shown in fig. 1, a hardware configuration diagram of an electronic device according to an embodiment of the present invention is provided.
The electronic device 1000 may be a smart device such as a Virtual Reality (VR) device, an Augmented Reality (AR) device, or a Mixed Reality (MR) device. Such as a virtual helmet.
In one example, the electronic device 1000 may be as shown in fig. 1, including a processor 1010, a memory 1020, an interface device 1030, a communication device 1040, a display device 1050, an input device 1060, and/or the like. Although the electronic device may also include a speaker, a microphone, and the like, these components are not relevant to the present invention and are omitted here.
The processor 1010 may be, for example, a central processing unit CPU, a microprocessor MCU, or the like. The memory 1020 includes, for example, a ROM (read only memory), a RAM (random access memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1030 includes, for example, a USB interface, a serial interface, an infrared interface, and the like. The communication device 1040 can perform wired or wireless communication, for example. The display device 1050 is, for example, a liquid crystal display, an LED display, a touch display, or the like. The input device 1060 includes, for example, a touch panel, a keyboard, a motion sensing input, and the like.
Although a number of devices are shown in fig. 1 for electronic device 1000, the present invention may relate to only some of the devices, for example, electronic device 1000 may relate to only memory 1020 and processor 1010.
In the above description, the skilled person can design the instructions according to the solutions provided in the present disclosure. How the instructions control the operation of the processor is well known in the art and will not be described in detail herein.
The electronic device shown in fig. 1 is merely illustrative and is in no way intended to limit the present disclosure, its application, or uses.
< electronic apparatus >
Referring to fig. 2, an electronic device provided by an embodiment of the invention is described.
The electronic device 2000 includes a processing unit 2010, and a stereo camera 2020, a switcher 2030 and an image processor 2040 electrically connected to the processing unit 2010, respectively, and an external interface 2050 is provided on a housing of the electronic device 2000.
The electronic device 2000 may be a smart device, such as a virtual helmet, an augmented reality device, or a mixed reality device.
The processing unit 2010 may be a central processing unit CPU, a microprocessor MCU or the like, such as a processor model APQ 8096.
The stereo camera 2020 may include at least two sub-cameras. For example, the stereo camera 2020 may be a binocular stereo camera including a left camera and a right camera. Such as a stereo camera module of type OV 7251.
The Switch 2030 may be a MIPI Switch (MIPI Switch) that is a mobile industry processor interface, for example, the Switch 2030 is a MIPI Switch with a model of NX3DV 642.
The image processor 2040 may be implemented by a Field Programmable Gate Array (FPGA). Such as a field programmable gate array of the type LFE 5U-45F.
The external interface 2050 may be a USB interface, such as USB2.0 or USB 3.0.
The stereo camera 2020 is used for photographing under the control of the processing unit 2010. Specifically, the processing unit 2010 is electrically connected to the stereo camera 2020, and the processing unit 2010 may transmit operating instructions to the stereo camera 2020 to control the operation of the stereo camera 2020. For example, a user triggers an instruction for starting the stereo camera to take a picture through a physical key or a virtual key of the virtual helmet, and the processor of the virtual helmet sends a picture taking instruction to the stereo camera in response to the instruction for starting the stereo camera to take a picture, so as to control the stereo camera to take a picture. For example, the virtual helmet is controlled by a mobile phone connected to the virtual helmet, and the processor of the virtual helmet sends a photographing instruction to the stereo camera in response to the instruction of the mobile phone to control the stereo camera to photograph. For example, when the virtual helmet is in communication with an external device, the processor also sends a clock synchronization signal to the stereo camera.
The image processor 2040 is configured to process images captured by the stereo camera 2020. Specifically, the image processor 2040 is electrically connected to the stereo camera 2020 and the processing unit 2010, respectively, and the image processor 2040 acquires an image captured by the stereo camera 2020, performs analysis processing on the image to obtain an operation result, and directly outputs or sends the operation result to the processing unit 2010. For example, an image of the stereo camera is processed by using a Field Programmable Gate Array (FPGA), an algorithm of the FPGA is configured, and the FPGA obtains an operation result according to the image photographed by the stereo camera and outputs the operation result.
The processing unit 2010 is configured to obtain a command through the external interface 2050, and control the switch 2030 to switch between the first path and the second path according to the command.
Wherein the first path communicates the stereo camera 2020 with the processing unit 2010 and the second path communicates the stereo camera 2020 with the image processor 2040.
The switch 2030 includes a first path and a second path, and when the switch 2030 is switched to the first path, the stereo camera 2020 and the processing unit 2010 are in communication, for example, a MIPI signal line of the stereo camera 2020 is in communication with a CSI interface of the processing unit 2010, wherein the MIPI signal line is a mobile industry processor interface signal line, and the CSI interface is a camera serial interface; when the switch 2030 is switched to the second path, the stereo camera 2020 and the image processor 2040 communicate.
In a specific example, the stereo camera 2020 is configured to take a picture of the test image card under control of the processing unit 2010 to obtain a test image;
the test image card, which may be a checkerboard card, is used to calibrate the electronic device, i.e., the stereo camera 2020 of the electronic device.
For example, when the stereo camera 2020 is a binocular stereo camera including a left camera and a right camera, the corresponding test images include test images obtained by photographing the test image card by the left camera and the right camera respectively.
In a specific example, when attempting to calibrate the electronic device, the command acquired by the processing unit 2010 through the external interface 2050 includes a first command, and the processing unit 2010 is configured to control the switch 2030 to switch to the first path according to the first command to acquire the test image through the first path;
the processing unit 2010 is further configured to provide the test image to the external calibration device through the external interface 2050, so that the external calibration device can calculate the calibration parameters according to the test image.
The external calibration device may be an upper computer, such as a single chip microcomputer.
For example, the electronic device 2000 is connected to the external calibration apparatus through a USB interface to implement bidirectional communication between the electronic device 2000 and the external calibration apparatus, the processing unit 2010 acquires a calibration command of the external calibration apparatus, and controls the switch 2030 to switch to the first path according to the calibration command, that is, the stereo camera 2020 communicates with the processing unit 2010 to acquire a test image captured by the stereo camera 2020 through the first path and provide the test image to the external calibration apparatus through the USB interface.
The external calibration device calculates calibration parameters according to the test image, and specifically, the external calibration device compares the obtained test image with the test image card to calculate the calibration parameters. For example, when the stereo camera is a binocular stereo camera, the external calibration device compares the test image photographed by the left camera and the test image photographed by the right camera with the test image card, respectively, and calculates calibration parameters. The calibration parameters include focal length, distortion parameters, translation vectors, deflection angles and the like of a camera of the stereo camera.
In a specific example, the processing unit 2010 is further configured to obtain calibration parameters from an external calibration device through the external interface 2050 and configure the calibration parameters to the image processor 2040. For example, the electronic device is connected with an external calibration device through a USB interface to realize bidirectional communication between the electronic device and the external calibration device, the calibration parameters are returned to the processor through the USB interface, and the processor performs configuration update on an algorithm of a Field Programmable Gate Array (FPGA) according to the calibration parameters.
In a specific example, when the electronic device performs regular photographing, the command acquired by the processing unit 2010 through the external interface 2050 further includes a second command, and the processing unit 2010 is configured to control the switch 2030 to switch to the second path according to the second command.
In this example, when the changeover switch 2030 is switched to the second path, the stereo camera 2020 is connected to the image processor 2040, and the image data acquired by the stereo camera 2020 is acquired by the second path image processor 2040 and subjected to arithmetic processing, thereby obtaining an arithmetic result.
The image processor 2040 may perform a calibration process on the data acquired by the stereo camera 2020 and obtain an operation result.
According to the electronic device provided by the embodiment of the invention, the processing unit of the electronic device controls the change-over switch to switch between the first channel and the second channel according to the acquired command, and when the electronic device is calibrated, the change-over switch is controlled to switch to the first channel so as to acquire the test image shot by the stereo camera through the first channel, and the test image is sent to the external calibration device through the external interface so that the external calibration device can calibrate the electronic device according to the test image. By adopting the scheme, the electronic equipment is not required to be calibrated by depending on a specific tool connected to the specific interface of the FPGA, the problem that static electricity failure is easily introduced due to the fact that an assembly gap caused by the specific interface of the FPGA is large is avoided, and normal use of a product is guaranteed.
In addition, the scheme does not adopt an FPGA specific interface for calibration, so that the use of the FPGA specific interface can be saved.
By adopting the scheme, the change-over switch is arranged in the electronic equipment after leaving the factory. After the electronic equipment leaves the factory, when needing to carry out the calibration, connect electronic equipment to host computer, control change over switch intercommunication simultaneously required route through external interface can, need not to dismantle electronic equipment and can easily carry out calibration operation, also reduced the complexity of the production line test of current electronic equipment simultaneously to manpower and calibration cost have been saved.
< calibration method of electronic device >
The electronic device has two modes, a calibration mode and a conventional photographing mode.
In a case that the electronic device is in a calibration mode, as shown in fig. 3, an embodiment of the present invention further provides a calibration method for an electronic device, including the following steps:
step S310, controlling the switch 2030 to switch to the first path according to the obtained command;
in step S320, the test image captured by the stereo camera 2020 is sent to the external calibration apparatus through the first path, so that the external calibration apparatus calibrates the electronic device according to the test image.
Specifically, under the condition that the electronic device is in the calibration mode, the electronic device controls the switch 2030 to switch to the first path according to the first command, that is, the processing unit 2010 of the electronic device is connected to the stereo camera 2020, a test image captured by the stereo camera 2020 is acquired through the first path, bidirectional communication between the electronic device and an external calibration device is realized through the setting of the external interface 2050, the electronic device transmits the test image to the external calibration device, the external calibration device performs test analysis on the test image, and returns a calibration parameter obtained through the test analysis to the electronic device, and the electronic device performs configuration update on an algorithm of the image processor 2040 of the electronic processing according to the calibration parameter, thereby realizing calibration of the electronic device.
In a case that the electronic device is in a conventional photographing mode, as shown in fig. 4, an embodiment of the present invention further provides an operating method of the electronic device, including the following steps:
step S410, controlling the switch 2030 to switch to the second path according to the obtained command;
and step S420, performing operation processing on the image acquired by the electronic equipment through the second channel to obtain and output an operation result.
Specifically, when the electronic device is in a normal photographing mode, the electronic device controls the switch 2030 to switch to the second path according to the second command, that is, the stereo camera 2020 of the electronic device is connected to the image processor 2040, an image captured by the stereo camera 2020 is sent to the image processor 2040 through the second path, and the image processor 2040 performs arithmetic processing according to the image captured by the stereo camera 2020, obtains an arithmetic result, and outputs the arithmetic result.
In this example, the electronic device is an electronic device that updates the algorithm configuration of the image processor through calibration processing before factory shipment, or an electronic device that needs to be calibrated due to a fault is an electronic device that updates the algorithm configuration of the image processor through calibration processing.
According to the calibration method provided by the embodiment of the invention, the processing unit controls the change-over switch to switch between the first channel and the second channel according to the acquired command, and controls the change-over switch to the first channel when the electronic equipment is calibrated, so that the test image shot by the stereo camera is acquired through the first channel, and the test image is sent to the external calibration device through the external interface, so that the external calibration device can calibrate the electronic equipment according to the test image; and under a conventional shooting mode, the change-over switch is controlled to be switched to the second channel, so that the images shot by the electronic equipment are processed and output. By adopting the scheme, the electronic equipment is not required to be calibrated by depending on a specific tool connected to the specific interface of the FPGA, the problem that static electricity failure is easily introduced due to the fact that an assembly gap caused by the specific interface of the FPGA is large is avoided, and normal use of a product is guaranteed.
In addition, the scheme does not adopt an FPGA specific interface for calibration, so that the use of the FPGA specific interface can be saved.
By adopting the scheme, the change-over switch is arranged in the electronic equipment after leaving the factory. After the electronic equipment leaves the factory, when needing to carry out the calibration, connect electronic equipment to host computer, control change over switch intercommunication simultaneously required route through external interface can, need not to dismantle electronic equipment and can easily carry out calibration operation, also reduced the complexity of the production line test of current electronic equipment simultaneously to manpower and calibration cost have been saved.
The calibration method provided in this embodiment corresponds to the electronic device provided in the foregoing embodiment, and the foregoing embodiment of the electronic device may be used to explain the calibration method in this embodiment, which is not repeated in this embodiment.
< computer storage Medium >
The embodiment of the invention also provides a computer storage medium, wherein the computer storage medium stores executable computer instructions, and when the executable computer instructions are executed by a processor, the calibration method provided by the calibration method embodiment can be realized.
Specifically, the foregoing embodiments of the calibration method can be used to explain the computer storage medium of this embodiment, and details are not repeated in the embodiments of the computer storage medium.
The above embodiments mainly focus on differences from other embodiments, but it should be clear to those skilled in the art that the above embodiments can be used alone or in combination with each other as needed.
The embodiments in the present disclosure are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from the other embodiments, but it should be clear to those skilled in the art that the embodiments described above can be used alone or in combination with each other as needed. In addition, for the device embodiment, since it corresponds to the method embodiment, the description is relatively simple, and for relevant points, refer to the description of the corresponding parts of the method embodiment. The system embodiments described above are merely illustrative, in that modules illustrated as separate components may or may not be physically separate.
The present invention may be an apparatus, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement various aspects of the present invention.
The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present invention may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, by software, and by a combination of software and hardware are equivalent. Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.