39 Augmented Reality

From Wikipedia: https://en.wikipedia.org/wiki/Augmented_reality

Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real-world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory.^[1][2] The overlaid sensory information can be constructive (i.e. additive to the natural environment), or destructive (i.e. masking of the natural environment).^[3] This experience is seamlessly interwoven with the physical world such that it is perceived as an immersive aspect of the real environment.^[3] In this way, augmented reality alters one’s ongoing perception of a real-world environment, whereas virtual reality completely replaces the user’s real-world environment with a simulated one.^[4][5] Augmented reality is related to two largely synonymous terms: mixed reality and computer-mediated reality.

The primary value of augmented reality is the manner in which components of the digital world blend into a person’s perception of the real world, not as a simple display of data, but through the integration of immersive sensations, which are perceived as natural parts of an environment. The earliest functional AR systems that provided immersive mixed reality experiences for users were invented in the early 1990s, starting with the Virtual Fixtures system developed at the U.S. Air Force’s Armstrong Laboratory in 1992.^[3][6][7] Commercial augmented reality experiences were first introduced in entertainment and gaming businesses. Subsequently, augmented reality applications have spanned commercial industries such as education, communications, medicine, and entertainment. In education, content may be accessed by scanning or viewing an image with a mobile device or by using markerless AR techniques.^[8][9] An example relevant to the construction industry is an AR helmet for construction workers which displays information about construction sites.

Augmented reality is used to enhance natural environments or situations and offer perceptually enriched experiences. With the help of advanced AR technologies (e.g. adding computer vision, incorporating AR cameras into smartphone applications and object recognition) the information about the surrounding real world of the user becomes interactive and digitally manipulated. Information about the environment and its objects is overlaid on the real world. This information can be virtual^{[10][11][12][13]} or real, e.g. seeing other real sensed or measured information such as electromagnetic radio waves overlaid in exact alignment with where they actually are in space.^[14][15][16] Augmented reality also has a lot of potential in the gathering and sharing of tacit knowledge. Augmentation techniques are typically performed in real time and in semantic contexts with environmental elements. Immersive perceptual information is sometimes combined with supplemental information like scores over a live video feed of a sporting event. This combines the benefits of both augmented reality technology and heads up display technology (HUD).

Applications

Augmented reality has many applications. First used for military, industrial, and medical applications, it has also been applied to commercial and entertainment areas.^[73]

Literature

In 2011, there were works using AR poetry made by ni_ka from the Sekai Camera in Japan, Tokyo. The rose of these works come from Paul Celan, “Die Niemandsrose“, and express the mourning of 3.11, 2011 Tōhoku earthquake and tsunami. ^{[74][75][76][77]}

Archaeology

AR can be used to aid archaeological research, by augmenting archaeological features onto the modern landscape, enabling archaeologists to formulate conclusions about site placement and configuration.^[78]

Another application given to AR in this field is the possibility for users to rebuild ruins, buildings, landscapes or even ancient characters as they formerly existed.^[79][80][81]

Architecture

AR can aid in visualizing building projects. Computer-generated images of a structure can be superimposed into a real life local view of a property before the physical building is constructed there; this was demonstrated publicly by Trimble Navigation in 2004. AR can also be employed within an architect’s work space, rendering into their view animated 3D visualizations of their 2D drawings. Architecture sight-seeing can be enhanced with AR applications allowing users viewing a building’s exterior to virtually see through its walls, viewing its interior objects and layout.^[82][83][84]

Visual art

AR technology has helped disabled individuals create visual art by using eye tracking to translate a user’s eye movements into drawings on a screen.^[85] An item such as a commemorative coin can be designed so that when scanned by an AR-enabled device it displays additional objects and layers of information that were not visible in a real world view of it.^[86][87] In 2013, L’Oreal used CrowdOptic technology to create an augmented reality at the seventh annual Luminato Festival in Toronto, Canada.^[31]

AR in visual art opens the possibility of multidimensional experiences and interpretations of reality. Augmenting people, objects, and landscapes is becoming an art form in itself. In 2011, artist Amir Bardaran’s Frenchising the Mona Lisa infiltrates Da Vinci’s painting using an AR mobile application called Junaio. Aim a Junaio loaded smartphone camera at any image of the Mona Lisa and watch as Leonardo’s subject places a scarf made of a French flag around her head.^[88] The AR app allows the user to train his or her smartphone on Da Vinci’s Mona Lisa and watch the mysterious Italian lady loosen her hair and wrap a French flag around her in the form a (currently banned) Islamic hijab.^[89]

Greeting cards

AR technology has been used in conjunction with greeting cards. They can be implemented with digital content which users are able to discover by viewing the illustrations with certain mobile applications or devices using augmented reality technology. The digital content could be 2D & 3D animations, standard video and 3D objects with which the users can interact.

In 2015, the Bulgarian startup iGreet developed its own AR technology and used it to make the first premade “live” greeting card. It looks like traditional paper card, but contains hidden digital content which only appears when users scan the greeting card with the iGreet app.^[90][91]

Commerce

AR can enhance product previews such as allowing a customer to view what’s inside a product’s packaging without opening it.^[92] AR can also be used as an aid in selecting products from a catalog or through a kiosk. Scanned images of products can activate views of additional content such as customization options and additional images of the product in its use.^[93][94] AR is used to integrate print and video marketing. Printed marketing material can be designed with certain “trigger” images that, when scanned by an AR enabled device using image recognition, activate a video version of the promotional material. A major difference between Augmented Reality and straight forward image recognition is that you can overlay multiple media at the same time in the view screen, such as social media share buttons, in-page video even audio and 3D objects. Traditional print only publications are using Augmented Reality to connect many different types of media.^{[95][96][97][98][99]}

Construction

With the continual improvements to GPS accuracy, businesses are able to use augmented reality to visualize georeferenced models of construction sites, underground structures, cables and pipes using mobile devices.^[100] Augmented reality is applied to present new projects, to solve on-site construction challenges, and to enhance promotional materials.^[101] Examples include the Daqri Smart Helmet, an Android-powered hard hat used to create augmented reality for the industrial worker, including visual instructions, real time alerts, and 3D mapping.^[102]

Following the Christchurch earthquake, the University of Canterbury released, CityViewAR, which enabled city planners and engineers to visualize buildings that were destroyed in the earthquake.^[103] Not only did this provide planners with tools to reference the previous cityscape, but it also served as a reminder to the magnitude of the devastation caused, as entire buildings were demolished.

Education

Augmented reality applications can complement a standard curriculum. Text, graphics, video and audio can be superimposed into a student’s real time environment. Textbooks, flashcards and other educational reading material can contain embedded “markers” that, when scanned by an AR device, produce supplementary information to the student rendered in a multimedia format.^{[104][105][106]}Students can participate interactively with computer generated simulations of historical events, exploring and learning details of each significant area of the event site.^[107] On higher education, there are some applications that can be used. For instance, Construct3D, a Studierstube system, allows students to learn mechanical engineering concepts, math or geometry. This is an active learning process in which students learn to learn with technology.^[108] AR can aid students in understanding chemistry by allowing them to visualize the spatial structure of a molecule and interact with a virtual model of it that appears, in a camera image, positioned at a marker held in their hand.^[109] It can also enable students of physiology to visualize different systems of the human body in three dimensions.^[110]Augmented reality technology also permits learning via remote collaboration, in which students and instructors not at the same physical location can share a common virtual learning environment populated by virtual objects and learning materials and interact with another within that setting.^[111]

This resource could also be of advantage in Primary School. Children can learn through experiences, and visuals can be used to help them learn. For instance, they can learn new knowledge about astronomy, which can be difficult to understand, and children might better understand the solar system when using AR devices and being able to see it in 3D. Further, learners could change the illustrations in their science books by using this resource.
For teaching anatomy, teachers could visualize bones and organs using augmented reality to display them on the body of a person.^{[citation needed]}

Mobile apps using augmented reality are emerging in the classroom. The mix of real life and virtual reality displayed by the apps using the mobile phone’s camera allows information to be manipulated and seen like never before. Many such apps have been designed to create a highly engaging environment and transform the learning experience. Examples of the mobile apps, that leverage augmented reality to aid learning, include SkyView for studying astronomy^[112] and AR Circuits for building simple electric circuits.^[113]

Emergency management / search and rescue

Augmented reality systems are used in public safety situations – from super storms to suspects at large. Two interesting articles fromEmergency Management magazine discuss the power of the technology for emergency management. The first is “Augmented Reality–Emerging Technology for Emergency Management” by Gerald Baron.^[114] Per Adam Crowe: “Technologies like augmented reality (ex: Google Glass) and the growing expectation of the public will continue to force professional emergency managers to radically shift when, where, and how technology is deployed before, during, and after disasters.”.^[115]

Another example, a search aircraft is looking for a lost hiker in rugged mountain terrain. Augmented reality systems provide aerial camera operators with a geographic awareness of forest road names and locations blended with the camera video. As a result, the camera operator is better able to search for the hiker knowing the geographic context of the camera image. Once found, the operator can more efficiently direct rescuers to the hiker’s location.^[116]

Everyday

Since the 1970s and early 1980s, Steve Mann has been developing technologies meant for everyday use i.e. “horizontal” across all applications rather than a specific “vertical” market. Examples include Mann’s “EyeTap Digital Eye Glass”, a general-purpose seeing aid that does dynamic-range management (HDR vision) and overlays, underlays, simultaneous augmentation and diminishment (e.g. diminishing the electric arc while looking at a welding torch).^[117]

Video games

Augmented reality allows video game players to experience digital game play in a real world environment. Companies and platforms like Niantic and LyteShot emerged as augmented reality gaming creators.^[118][119] Niantic is notable for releasing the record-breakingPokémon Go game.^[120]

Industrial design

AR can help industrial designers experience a product’s design and operation before completion. Volkswagen uses AR for comparing calculated and actual crash test imagery.^[121] AR can be used to visualize and modify a car body structure and engine layout. AR can also be used to compare digital mock-ups with physical mock-ups for finding discrepancies between them.^[122][123]

Medical

Since 2005, a device that films subcutaneous veins, processes and projects the image of the veins onto the skin has been used to locate veins. This device is called a near-infrared vein finder.^[124][125]

Augmented Reality can provide the surgeon with information, which are otherwise hidden, such as showing the heartbeat rate, the blood pressure, the state of the patient’s organ, etc. AR can be used to let a doctor look inside a patient by combining one source of images such as an X-ray with another such as video.

Examples include a virtual X-ray view based on prior tomography or on real time images from ultrasound and confocal microscopy probes,^[126] visualizing the position of a tumor in the video of an endoscope,^[127] or radiation exposure risks from X-ray imaging devices.^[128][129] AR can enhance viewing a fetus inside a mother’s womb.^[130] It has been also used for cockroach phobia treatment.^[131] Also, patients wearing augmented reality glasses can be reminded to take medications.^[132]

Beauty

In 2014 the company L’Oreal Paris started developing a smartphone and tablet application called “Makeup Genius”, which lets users try out make-up and beauty styles utilizing the front-facing camera of the endpoint and its display.^[133]

Spatial immersion and interaction

Augmented reality applications, running on handheld devices utilized as virtual reality headsets, can also digitalize human presence in space and provide a computer generated model of them, in a virtual space where they can interact and perform various actions. Such capabilities are demonstrated by “project Anywhere” developed by a post graduate student at ETH Zurich, which was dubbed as an “out-of-body experience” [6][7][8].

Military

In combat, AR can serve as a networked communication system that renders useful battlefield data onto a soldier’s goggles in real time. From the soldier’s viewpoint, people and various objects can be marked with special indicators to warn of potential dangers. Virtual maps and 360° view camera imaging can also be rendered to aid a soldier’s navigation and battlefield perspective, and this can be transmitted to military leaders at a remote command center.^[134]

An interesting application of AR occurred when Rockwell International created video map overlays of satellite and orbital debris tracks to aid in space observations at Air Force Maui Optical System. In their 1993 paper “Debris Correlation Using the Rockwell WorldView System” the authors describe the use of map overlays applied to video from space surveillance telescopes. The map overlays indicated the trajectories of various objects in geographic coordinates. This allowed telescope operators to identify satellites, and also to identify – and catalog – potentially dangerous space debris.^[135]

Starting in 2003 the US Army integrated the SmartCam3D augmented reality system into the Shadow Unmanned Aerial System to aid sensor operators using telescopic cameras to locate people or points of interest. The system combined both fixed geographic information including street names, points of interest, airports and railroads with live video from the camera system. The system offered “picture in picture” mode that allows the system to show a synthetic view of the area surrounding the camera’s field of view. This helps solve a problem in which the field of view is so narrow that it excludes important context, as if “looking through a soda straw”. The system displays real-time friend/foe/neutral location markers blended with live video, providing the operator with improved situation awareness.

Researchers at USAF Research Lab (Calhoun, Draper et al.) found an approximately two-fold increase in the speed at which UAV sensor operators found points of interest using this technology.^[136] This ability to maintain geographic awareness quantitatively enhances mission efficiency. The system is in use on the US Army RQ-7 Shadow and the MQ-1C Gray Eagle Unmanned Aerial Systems.

Navigation

AR can augment the effectiveness of navigation devices. Information can be displayed on an automobile’s windshield indicating destination directions and meter, weather, terrain, road conditions and traffic information as well as alerts to potential hazards in their path.^{[137][138][139]} Aboard maritime vessels, AR can allow bridge watch-standers to continuously monitor important information such as a ship’s heading and speed while moving throughout the bridge or performing other tasks.^[140]

The NASA X-38 was flown using a Hybrid Synthetic Vision system that overlaid map data on video to provide enhanced navigation for the spacecraft during flight tests from 1998 to 2002. It used the LandForm software and was useful for times of limited visibility, including an instance when the video camera window frosted over leaving astronauts to rely on the map overlays.^[141] The LandForm software was also test flown at the Army Yuma Proving Ground in 1999. In the photo at right one can see the map markers indicating runways, air traffic control tower, taxiways, and hangars overlaid on the video.^[142]

Office workplace

AR can help facilitate collaboration among distributed team members in a work force via conferences with real and virtual participants. AR tasks can include brainstorming and discussion meetings utilizing common visualization via touch screen tables, interactive digital whiteboards, shared design spaces, and distributed control rooms.^{[143][144][145]}

Sports and entertainment

AR has become common in sports telecasting. Sports and entertainment venues are provided with see-through and overlay augmentation through tracked camera feeds for enhanced viewing by the audience. Examples include the yellow “first down” line seen in television broadcasts of American football games showing the line the offensive team must cross to receive a first down. AR is also used in association with football and other sporting events to show commercial advertisements overlaid onto the view of the playing area. Sections of rugby fields and cricket pitches also display sponsored images. Swimming telecasts often add a line across the lanes to indicate the position of the current record holder as a race proceeds to allow viewers to compare the current race to the best performance. Other examples include hockey puck tracking and annotations of racing car performance and snooker ball trajectories.^[58][146]

AR can enhance concert and theater performances. For example, artists can allow listeners to augment their listening experience by adding their performance to that of other bands/groups of users.^{[147][148][149]}

The gaming industry has benefited a lot from the development of this technology. A number of games have been developed for prepared indoor environments. Early AR games also include AR air hockey, collaborative combat against virtual enemies, and AR-enhanced pool games. A significant number of games incorporate AR in them and the introduction of the smartphone has made a bigger impact.^[150][151]

Task support

Complex tasks such as assembly, maintenance, and surgery can be simplified by inserting additional information into the field of view. For example, labels can be displayed on parts of a system to clarify operating instructions for a mechanic who is performing maintenance on the system.^[152][153] Assembly lines gain many benefits from the usage of AR. In addition to Boeing, BMW and Volkswagen are known for incorporating this technology in their assembly line to improve their manufacturing and assembly processes.^{[154][155][156]} Big machines are difficult to maintain because of the multiple layers or structures they have. With the use of AR the workers can complete their job in a much easier way because AR permits them to look through the machine as if it was with x-ray, pointing them to the problem right away.^[157]

Television

Weather visualizations were the first application of augmented reality to television. It has now become common in weathercasting to display full motion video of images captured in real-time from multiple cameras and other imaging devices. Coupled with 3D graphics symbols and mapped to a common virtual geo-space model, these animated visualizations constitute the first true application of AR to TV.

Augmented reality has also become common in sports telecasting. Sports and entertainment venues are provided with see-through and overlay augmentation through tracked camera feeds for enhanced viewing by the audience. Examples include the yellow “first down” line seen in television broadcasts of American football games showing the line the offensive team must cross to receive a first down. AR is also used in association with football and other sporting events to show commercial advertisements overlaid onto the view of the playing area. Sections of rugby fields and cricket pitches also display sponsored images. Swimming telecasts often add a line across the lanes to indicate the position of the current record holder as a race proceeds to allow viewers to compare the current race to the best performance. Other examples include hockey puck tracking and annotations of racing car performance and snooker ball trajectories.^[158][159]

Augmented reality is starting to allow Next Generation TV viewers to interact with the programs they are watching. They can place objects into an existing program and interact with these objects, such as moving them around. Avatars of real persons in real time who are also watching the same program.^[160]

Tourism and sightseeing

Augmented reality applications can enhance a user’s experience when traveling by providing real time informational displays regarding a location and its features, including comments made by previous visitors of the site. AR applications allow tourists to experience simulations of historical events, places and objects by rendering them into their current view of a landscape.^{[161][162][163]} AR applications can also present location information by audio, announcing features of interest at a particular site as they become visible to the user.^{[164][165][166]}

Translation

AR systems can interpret foreign text on signs and menus and, in a user’s augmented view, re-display the text in the user’s language. Spoken words of a foreign language can be translated and displayed in a user’s view as printed subtitles.^{[167][168][169]}