2021 is the year of the metaverse. Metaverse-related issues are attracting the attention of many companies. These companies are racking their brains to think of ways to make their products related to metaverse. Nowadays, the various applications of metaverse have brought dramatic changes to the fields of games, education, media industry, etc. The author is also curious about what could happen when the metaverse is combined with the automotive industry. Through patent search and analysis, this article will provide a glimpse of the answers to this question.  

Patent Search Description

According to Wikipedia, the metaverse is a hypothetical iteration of the Internet as a single, universal and immersive virtual world that is facilitated by the use of virtual reality (VR) and augmented reality (AR) headsets. Therefore, the search scope for metaverse x future cars related patents is an intersection of automotive technology and Virtual Reality (VR), Augmented Reality (AR), or Extended Reality (XR) technologies, and is limited to the patents filed after 2010. The patents within such scope are about 2,300 patent families, as shown in the below table.   Patent Filing Trend Fig.1 illustrates the change number of patents of metaverse x future cars filed between 2010 and 2022. It is shown that, when the development of head mounted devices started in 2014, the number of related patents also increased. In 2019, head mounted devices became less popular, so the number of patents declined. Patents filed after 2021 have not yet been fully disclosed[1], so the author predicts that the number of patents to be disclosed after 2021 will rise again because the metaverse issue has driven the development of VR and AR technologies over the past two years.  

Fig.1 Filing trend of metaverse x future cars related patents

 

Major Patent Owners

  Fig.2 shows the major patent owners of metaverse x future cars related patents. Automakers such as Toyota and Honda have the most patents of metaverse x future cars. Car parts suppliers (such as Magna) and service platforms (such as UBER) are also gradually deploying related patents, as well as ICT companies (such as Samsung and Microsoft) that provide hardware and software solutions for VR and AR technologies.

Fig.2 Major patent owners of metaverse x future cars related patents

 

Hardware

  Metaverse x future cars related patents can be categorized into two major groups of technologies, hardware and user scenarios. Fig. 3 illustrates the hardware described in metaverse x future cars related patents, including display devices and sensing devices. The display devices are used to display virtual and reality images, including head mounted displays, head up displays, in-car displays, etc. Sensing devices are used to detect environment or driver’s information, including vehicle exterior sensing and vehicle interior sensing devices. Description of patents will be given as examples of the hardware related to metaverse x future cars.

Fig. 3 Hardware described in metaverse x future cars related patents

 

1. Display device - Head Mounted Display (HMD)

  Head mounded displays include head mounted types and eyeglass types, providing drivers or passengers virtual images when using vehicles. Taking AUDI’s patent US11181739 as an example, referred to Fig. 4, it is shown that a driver is wearing an HMD. The HMD can display a virtual vehicle interior to allow the driver to experience the interior of the car as seen in various vehicles.

Fig. 4 Drawing of AUDI’s patent US11181739

 

2. Display device - Head Up Display (HUD)

  The difference between head-up displays and conventional in-car displays is that the head-up displays can project images onto the windshield or side window glass, making it easier for the driver to simultaneously see driving information and the outside view of the car with a straightforward view, avoiding the distraction caused by the conventional in-car displays that require a change of view angle. Taking Samsung’s patent US20190317325 as an example, referred to Fig. 5 and Fig. 6, an image output device 110 projects an image through a reflector 130 onto an optical film 140 attached to the windshield 150 of a vehicle; through the reflection effect of the optical film 140, the driver can directly see the virtual image at the view angle of the windshield. As shown in Samsung's patent US20210207971, the virtual image can display navigation information so that the driver can quickly be aware of the driving path and avoid distraction by looking down at a conventional navigation device.

Fig. 5 Drawing of Samsung’s patent US20190317325

 

Fig. 6 Drawing of Samsung’s patent US20210207971

 

3. Display device -rearview mirror display

  A rearview mirror display is to add a display region in a rear mirror or replace the traditional rear mirror with a display, so that the rearview mirror can not only display the image of cars coming from behind, but also can display augmented images according to the driving needs. Taking PayPal’s patent US10088676 as an example, referred to Fig. 7, the rear mirror 302 of a driver shows a second car 304 coming from behind, but a third car 306 is blocked by the second car 304 and cannot be shown in the rear mirror 302; then a virtual image 308 can be used to add the third car 306 in the rear mirror 302, so that the rear mirror 302 of the driver’s car shows the third car 306 blocked by the rear car 304 to help the driver being aware of the oncoming car from behind.

Fig. 7 Drawing of PayPal’s US10088676

 

4. Other display devices

  Besides HMD, HUD, and rearview mirror display, the display devices described metaverse x future cars related patents also include CAVE (Cave Automatic Virtual Environment), in-car displays, smart phones, tablets, etc., to allow users to access metaverse technology while using the vehicle.  

5. Sensing device

  Besides display devices, metaverse x future cars related patents also involve many sensing devices, including vehicle exterior and interior sensing devices. The vehicle exterior sensing devices are used to detect objects outside vehicles, such as other vehicles, pedestrians, and traffic signs so that drivers are aware in a more acute manner of road-related information. Vehicle interior sensing devices are mainly used to detect the status of drivers or passengers, such as physiological information, gestures, and body orientations so that virtual images presented to the drivers or passengers can be controlled accordingly.   Taking UBER’s patent US9922466 as an example, the vehicle can include head tracking sensors 402 and hand and body tracking sensors 403. Head tracking sensors 402 monitor a rider's head, eyes, and ears relative to the environment for the virtual reality system, both in terms of position and orientation so that a virtual reality system can determine the direction toward which the rider is looking at. Hand and body tracking sensors 403 can monitor the body position, personal object volumes (e.g., a briefcase, purse, or bag), and surrounding free space of each rider in the vehicle. The vehicle can be equipped with external sensors such as mapping sensors 412, audio sensors 414, and positioning sensors 415 to detect external environment and positioning the vehicle.    

Fig. 8 Drawing of UBER’s patent US9922466

  From the aforementioned hardware involved in the metaverse x future cars related patents, it is necessary to improve display and sensing technologies on top of the traditional vehicle hardware in order to apply the metaverse concept to the automotive field. Although there was no mention of networking devices and vehicle CPUs in the previous section, to better connect and control these hardware, networking and CPU technology will inevitably be upgraded accordingly. If companies with relevant hardware technology can respond to this demand and provide suitable products, it might be a great opportunity to increase the value of their products. In addition to the hardware technology, the metaverse x future cars related patents focus more on the application of metaverse technology to various car use cases, which will be introduced in the next article.   [1] Invention patents are published eighteen months after their filing date.   Author: Jean Chou Taiwan Patent Attorney / Certified Valuation Analyst / Co-Chair of Business IP Management Committee, TWPAA / Vice President of WISPRO Technology Consulting Corporation Email: jeanchou@wispro.com