The VR180-3D format has been used for immersive media for several years. In this article, I look at new patent applications for stereo lenses from Canon with a different field of view and their potential impact on immersive media for virtual reality.
By Daniel Pohl
In the VR180-3D format, the environment is recorded in a hemisphere (180°× 180°) for each eye. In virtual reality, this format is suitable for high immersion, as head tracking with a simultaneous stereoscopic view brings the image very close to the experience of a real scene. But why exactly 180 degrees?
What does VR180 refer to?
Without virtual reality, stereoscopic VR180 makes not much sense. In VR, however, the format blossoms perfectly to provide both a 3D view and enough viewing angles to see other areas of the image with tracked head movements. Accordingly, the first VR180 cameras, such as Lucidcam, Z-Cam K1 Pro and the Lenovo Mirage Camera, came onto the market in 2017 and 2018 with the emergence of better VR headsets.
The VR180 images are recorded without any stitching, as is the case with 360° cameras. This avoids any stitching artifacts and inconsistencies such as different lighting of the individual images or problems with dynamic objects, as it can happen with 360-degree drone shots.
The images from VR180 cameras taken with round fisheye lenses are usually saved in the manufacturer-independent equirectangular format or converted afterward using tools. This is comparable to transferring a round globe into a rectangular format.
VR180 photo with Canon EOS R5, 5.2mm Dual Fisheye. Above: Image in fisheye format (left/right reversed due to lens). Below: Image in equirectangular format. | Picture: Daniel Pohl
If the equirectangular 180-degree 3D photo is split into two parts, one half of it will be a square that accurately depicts the 180°×180° hemisphere. In order to prevent black borders and enable smooth head movements in VR, it is advisable to fill the entire 180 degrees with content.
Why not consider angles of 190° or 200°? Such possibilities have been explored. For instance, the Z-Cam K2Pro provides an option with 200° fisheye lenses, enabling a view of the area behind the camera as well. Nevertheless, when observing the edges at 180°, the neighboring lens of the stereo camera will unavoidably become visible in one eye.
If the left and right eye receive significantly different and unrealistic information in 3D perception, it can cause discomfort, eye strain, and motion sickness. However, the VR180 Canon 5.2mm Dual Fisheye lens used in modern VR image viewers like immerGallery reduces this issue by cropping out a significant portion of the edges on both sides of the photos taken with this camera.
Close-up view of the lens adjacent to the Canon 5.2mm lens captured in a photo by Daniel Pohl.
Hence, the significance of lenses exceeding 180° for virtual reality is not definite. Upon comparing the fisheye images with the equirectangular version mentioned above, one can observe that the original fisheye image captures more content. The lens’s actual capacity can reach nearly 190°. However, displaying these extra pixels in VR does not provide any additional value due to the presence of adjacent lenses in the image and the diminishing fisheye quality towards the edges caused by chromatic aberrations. Furthermore, the 3D effect also diminishes towards the edges regardless.
Why 180° may currently be too much
While it is enjoyable to have the ability to move your head within virtual reality, there are challenges involved. Achieving detailed image data in a 180°× 180° space requires extremely high resolutions. The initial VR180 cameras in 2018 had approximately 6000 pixels horizontally and 3000 pixels vertically. While this may appear impressive in a Meta Quest 3 headset in terms of stereoscopic effect, the clarity is unfortunately somewhat blurred. However, utilizing 8K photo resolutions, like those provided by the Canon EOS R5 with a 5.2 mm lens, significantly improves image sharpness compared to previous options.
However, the 8K VR180 photos taken with this setup do not have the same sharpness as content displayed on a Quest 3 today. We can compare it to 360-degree images captured by drones. Drones capture multiple high-resolution images while rotating, resulting in a stitched image with several hundred megapixels. These images are not directly loaded into the headset at this resolution. Instead, they are scaled down to a resolution that is similar to the 8K-VR180 photos in terms of pixel count. Despite having the same number of pixels per degree of field of view, the downscaled images appear significantly more detailed and sharper.
Why is this the case? To begin, it is evident from the fisheye image above that not all of the pixels on the EOS R5 camera sensor (with 45 megapixels) are utilized, as the image of the two circular fisheyes needs to fit within the square camera sensor. The fisheye image already exhibits some distortions caused by the functioning of fisheye lenses. When converting it to the equirectangular format, pixels must be interpolated, resulting in a certain level of blurring, as the pixels are no longer used in precise accordance with their scanned pattern. The final equirectangular format also contains certain distortions. It is impossible to display a (half) sphere in a rectangle without experiencing distortion.
Why does the image captured by the drone appear much clearer when reduced in size? If advanced algorithms are utilized, a significant amount of data is preserved within the subpixels during the scaling process, resulting in effective anti-aliasing or super sampling from a rendering standpoint. Consequently, this enhances the perception of sharpness.
Is VR180 lacking in sharpness? What should be done in this situation?
What options remain if a VR camera equipped with a high-quality 8K professional camera body and an L lens fails to produce sharp results for VR180? The next potential solution would involve enhancing the sensor resolution in the camera bodies. Nevertheless, this approach would further escalate the cost of the products. While a horizontal resolution of 10K might be imaginable in the field of photography, creating a camera capable of capturing 10K video at 60 fps or more seems unattainable without incurring exorbitant expenses, particularly by 2024.
Canon has demonstrated prototypes of a new dual fisheye lens designed for camera bodies with APS-C sensors, which are more affordable than full-frame sensors. However, the highest resolution currently available for APS-C camera bodies is 32 megapixels in the EOS R7, while the full-frame EOS R5 offers 45 megapixels. VR180 photos with a horizontal resolution of approximately 7K should suffice for most personal users, but there will be a decrease in sharpness compared to the R5. So, what is the next step?
One dimension that has not yet been considered in cameras for immersive media is 180° × 180°. Of course, the aim is to remain well above the experience of a 3D movie theater or the old 3D cameras for “normal 3D” (approx. 70° field of view) and continue to support head movement in VR for immersion. However, it is conceivable that a certain reduction in the field of view is acceptable. This would mean that less of the surroundings would be projected on the same sensor, resulting in a sharper image for our VR photos.
VR150? VR135?
At the IEEE VR 2024 scientific conference, immerVR and the University of Würzburg presented a research project that showcased the combination of two smartphones. This combination resulted in the creation of a VR123 camera capable of capturing a stereoscopic 123° field of view. The purpose of this demonstration was to illustrate to smartphone manufacturers that it is possible to utilize modern smartphones for capturing content beyond standard 3D photos and videos with limited viewing angles, similar to the capabilities of the iPhone 15 Pro.
Using ultra-wide-angle lenses, it is possible to create immersive media with certain head rotation capabilities in VR. In this particular project, the head rotation angle was limited to 123 degrees due to camera interface restrictions. However, smartphones like the realme GT2 Pro already have integrated ultra-wide-angle cameras with a wider angle of 150 degrees. It will be interesting to see if further advancements will be made in this area.
![[paper.jpg] Forschungsarbeit mit zwei Smartphones zum Erzeugen von VR123 bzw. VR150-Medien.](https://virtualrealityheadsets.info/wp-content/uploads/2024/05/what-about-vr150-and-vr135-is-stereo-vr180-the-standard-for-immersive-media-3.jpg)
Research work with two smartphones to create VR123 and VR150 media. | Image: Scheuerpflug et al.
Canon has recently submitted two noteworthy patent applications. It is essential to remember that a patent application does not guarantee the development of a product and may solely serve as experimental designs. Specifically, CanonRumors website discusses patent applications for lenses informally known as Canon RF-S 3.2mm F2.8 Dual Fisheye and Canon RF-S 3.8mm F3.5 Dual Fisheye.
These values pertain to the APS-C sensor. The lenses have a described half-angle of view of 79.1° and 72°. When doubled, they result in approximately 158° and 144°. Unlike VR180, the neighboring lens is not a problem in this case. Nevertheless, as evident from the fisheye image above, the lens’s optical shortcomings at the edges suggest that the more extreme outer areas may not be utilized here either. This implies that the two stereo lenses could potentially provide the image content equivalent to VR150 and VR135 in the converted, equirectangular format.
What implications does this have for creators of immersive media? The reduced field of view eliminates common newbie mistakes where your own body parts, such as feet or fingers, end up in the frame. This is a scenario that most photographers using 180°×180° have likely experienced. Additionally, the use of a tripod becomes more convenient. In the case of VR180, the tripod legs would always be visible in the shot unless a longer rail is attached to the tripod.
The improved focus will allow for capturing the primary subject, such as a person in the center of the image, with even greater clarity than with VR180. However, it will no longer be as effective to stand near a tall building and frame a shot that requires viewers to tilt their heads upwards to see everything. In such cases, the vertical limitation of approximately 135° will come into play too soon. Consequently, there are still valid reasons to utilize a complete 180° setup and its smaller variations.
How do VR150 and VR135 appear?
I simulated the field of view restriction at 150° and 135° compared to the normal VR180 using image data from two original Canon VR180 EOS R5 5.2mm photos. This test is specifically for comparing the field of view impressions, not the sharpness, since all images were generated from the same original VR180 photo in 8K. Please note that due to distortions in the equirectangular format, the 2D impression shown in this article may differ from the VR experience.
I have therefore made the comparison photos of VR180, VR150 and VR135 available for download as a .zip file so that you can view them yourself in VR. The easiest way to do this is via the immerGallery app, where you can download the images at immervr://vr150. To do this, go in the app to Download, Custom URL, set the drop-down to immervr and then simply type vr150.
A VR180-3D image with the main object in the center of the photo. From top to bottom: 180°,150°, 135°. | Bild: Daniel Pohl
A VR180-3D image with the main object in the center of the photo. From top to bottom: 180°,150°, 135°. | Picture: Daniel Pohl
A three-dimensional VR180 image capturing a close-up view of a tall object, with view angles of 180°, 150°, and 135° from top to bottom. The image has been captured by Daniel Pohl.
When I use the VR headset to view VR180, VR150, and VR135, I personally feel that it is manageable to have a restricted angle of 150° if it enhances the image sharpness. With a 135° angle, the entire visual field is still covered, but minor movements of the head serve as a reminder that you are viewing an image rather than actually being present in the displayed world. Based on my initial observations, I would therefore choose the 150° option as my preference.
Summary
Immersive media can convey impressive VR experiences. However, VR180 still lacks the necessary sharpness of the generated content, even with more professional camera setups in the $5,000 range. It would also be helpful for the VR market if it were possible to create sharp and immersive 3D content more cost-effectively.
One possible alternative is to limit the field of view of immersive 3D camera lenses slightly. However, this restriction should still allow for some head movement to enhance immersion, and the edges of the field of view should be fully filled with image data. This compromise could be a viable solution to improve image sharpness in Virtual Reality (VR). It is anticipated that VR users will eventually demand this upgrade in headsets with higher resolutions, like the Apple Vision Pro, beyond the capabilities of the Quest 3.
According to rumors, Canon will hold a major presentation of new products in the week of May 13, 2024. We are excited and hope to see more products in the 3D and VR sector. Perhaps there will also finally be news in the area of the foldable consumer camera for VR180 and 360° 2D.
The author, Daniel Pohl, is CEO and founder of immerVR GmbH. There, Daniel works daily on innovations in the field of immersive media, mostly in the area of VR180 stereo photography. With his app immerGallery, you can experience immersive photo galleries with voice-overs and background music in various VR formats on Meta Quest headsets – also together with friends in multiplayer.
