Augmented Reality

U of T and Autodesk researchers design 2D-3D software for augmented reality drawing

noreen.rasbach — Fri, 20 Apr 2018 04:00:00 +0000

U of T and Autodesk researchers design 2D-3D software for augmented reality drawing noreen.rasbach Fri, 04/20/2018 - 00:00

Cutline

Tovi Grossman, a researcher at Autodesk, and PhD student Rahul Arora, are part of a research team developing 2D-3D software for augmented reality drawing (photo by Ryan Perez)

Nina Haikara

Topic

City & Culture

Alumni

Augmented Reality

Computer Science

Faculty of Arts & Science

Graduate Students

Research & Innovation

Technology

Prehistoric cave drawings date back about 40,000 years. Ancient Egyptians used papyrus for paper. Artistic drawing flourished during the European Renaissance starting in the 14^th century.

Now, University of Toronto computer scientists and Autodesk researchers are looking to take artistic vision from 2D to 3D with augmented reality technology.

“We tried out systems like Tilt Brush by Google, and we noticed it was extremely hard to draw in the air,” says Rahul Arora, a PhD student in the department of computer science. “One reason why is, we’ve been drawing on 2D paper. How can we transition those skills to the 3D world?”

Arora and Autodesk researchers Rubaiat Habib Kazi, Tovi Grossman – who joins the department computer science as an assistant professor this summer – George Fitzmaurice and Karan Singh, a U of T professor of computer science, created SymbiosisSketch, software for combining 2D and 3D sketching for detailed design of 3D objects.

Their research will be presented at the Association of Computing Machinery’s international conference on human factors in computing systems (ACM CHI) in Montreal beginning this weekend.

SymbiosisSketch uses a tablet to support the 2D action of drawing on a surface. An augmented reality headset, Microsoft HoloLens, brings the drawing to life in a 3D augmented view.

“You can think of the tablet as a camera you position in the air, and you draw within its field of view. So you're always creating 3D strokes even when you're drawing on the tablet," says Arora. "If you draw a line, you'll see it in real time on that surface, in the air. It’s like a drawing window you can move around in 3D space.”

Arora, who worked on the project in part during an internship at Autodesk Research, says the canvas size can increase, to draw very large objects, like cars or buildings, or decrease, to zoom in on drawing finer details, like a door handle.

The joint research is another example of the longstanding partnership between Autodesk, the 3D design, engineering and entertainment software powerhouse, and U of T computer science’s Dynamic Graphics Project lab.

Arora says future applications of SymbiosisSketch could include use in animation and storyboarding, but for now they’re exploring state-of-the-art 3D sketching and fabrication.

“I'm currently working on a fabrication project," he says. "Let’s say you have a 3D sketch – what if you want to manufacture it to be strong and lightweight? How can we use some physical simulation to guide you?”

A Captain America figure battles an augmented reality drawing of an airplane (image courtesy of the researchers)

There are of course limitations to the current hardware technology, which lends itself well to a lab environment. A bulky motion capture system consisting of six ceiling-mounted infrared cameras tracks the pen’s movement. Also, AR hardware needs better speed and accuracy.

But all of that is just a matter of time.

“As hardware improves, research on the software side moves forward as well,” says Arora.

“This project is really looking forward in the future.”

French cosmetics giant buys U of T 'beauty tech' startup

Christopher.Sorensen — Fri, 16 Mar 2018 20:10:24 +0000

French cosmetics giant buys U of T 'beauty tech' startup

Christopher.Sorensen Fri, 03/16/2018 - 16:10

Cutline

Parham Aarabi, an associate professor in the Faculty of Applied Science & Engineering, founded ModiFace 11 years ago. The startup's widely used technology allows consumers to virtually try on makeup and other beauty products (photo by Johnny Guatto)

Chris Sorensen

Topic

Global Lens

Artificial Intelligence

Augmented Reality

Computer Science

Entrepreneurship

Faculty of Applied Science & Engineering

Research & Innovation

Startups

Subheadline

“It solves a key problem the industry has had for over 100 years”

A University of Toronto startup that lets consumers virtually try on makeup, hair and other beauty products has been scooped up by the world’s biggest cosmetics company – a move that could result in Toronto becoming a hub for “beauty tech” research.

France’s L’Oréal said Friday it was buying ModiFace, which applies augmented reality and artificial intelligence to the beauty industry, for an undisclosed sum.

The deal was touted as a first for the 109-year-old cosmetics giant, which owns such popular brands as Maybelline, Lancôme, Kiehl’s and Shu Uemura.

“Beauty companies usually buy other beauty companies,” said Parham Aarabi, the founder and CEO of ModiFace and an associate professor in the department of electrical and computer engineering in the Faculty of Applied Science & Engineering.

“This will be the first tech company under L’Oréal.”

Aarabi founded ModiFace 11 years ago after he realized his research into computer vision and facial tracking could be applied to the cosmetics industry, allowing people to see what they look like wearing different shades of makeup and other beauty products.

He said the company’s first customer was the maker of Botox, which reached out to him after learning about a lip-tracking project that involved the addition of a “skin-clearing” effect.

“They came to me asking if we could create a simulation tool,” Aarabi said.

Read about the deal at CNN

Read about the deal at the Globe and Mail

ModiFace’s technology, now even more sophisticated, has since been adopted by more than 100 brands. The startup uses live video to track facial features like eyes and lips and then, based on those features, renders different makeup, hair or skin effects.

Aarabi said the company also uses artificial intelligence, or AI, to “understand both faces and consumers in terms of how they interact and the shades they try on.”

L’Oréal's purchase of ModiFace demonstrates how important technology has become to the US$460 billion global cosmetics industry. L’Oréal, for example, generated about eight per cent of its sales online last year, compared to just five per cent two years earlier, according to Reuters.

Lubomira Rochet, L’Oréal's chief digital officer, said in a statement the acquisition of ModiFace will “support the reinvention of the beauty experience around innovative services to help our customers.”

Master's in applied computing student Cheng Chang demonstrates ModiFace at a U of T computer science research event in November (photo by Paul Hillier)

Aarabi, who will remain ModiFace’s CEO, said ModiFace will continue to be based in Toronto – “on a corner of the U of T campus” – after the deal is completed. He added 60 of the company’s 70 employees, including researchers and engineers, have a connection to U of T, and that the company plans to tap the university’s vast pool of engineering and computer science talent as it continues to grow under its new owner. ModiFace said last year it was investing $4 million in new undergraduate and graduate internships, as well as support for U of T engineering research.

“U of T is one of the world’s most prominent research institutions, especially when it comes to AI and computer engineering – and that’s been key for us,” Aarabi said.

“So as we look to 200 and maybe 1,000 engineers down the road, I think U of T will be critical as part of that for us.”

While so-called “beauty tech” may not sound as potentially disruptive as self-driving cars or AI-powered lawyers, Aarabi said the sector, and the vast amount of money that underpins it, could play a significant role in contributing to Toronto’s emergence as a global research and innovation hub.

“It’s pretty established now in the beauty industry that AR is a key technology that every brand needs to have,” Aarabi said, adding that consumers place a high value on trying on cosmetics products before plunking down cash to buy them.

“It solves a key problem the industry has had for over 100 years.”

Learn more about U of T Entrepreneurship

What's it like to develop Google’s latest AR technology? Ask this U of T alumnus

noreen.rasbach — Thu, 18 Jan 2018 17:31:39 +0000

What's it like to develop Google’s latest AR technology? Ask this U of T alumnus

noreen.rasbach Thu, 01/18/2018 - 12:31

Cutline

Alumnus Ivan Neulander gives students a technical overview of the Google’s augmented reality (AR) stickers at the Department of Computer Science Innovation Lab (photo by Ryan Perez)

Topic

Department of Computer Science Innovation Lab

Research and Innovation

After completing his undergraduate and master’s degree in computer graphics at the University of Toronto, Ivan Neulander left his Toronto home to take a job with a visual effects company based in Los Angeles.

“Back in the late 90s, California was really the place to be for doing visual effects,” says the department of computer science graduate. “It’s become a lot more globalized now. Rhythm and Hues was the first one to make me a job offer, so the rest is history, as they say.”

The industry has seen swift change in the 20 years since. After 15 years with the company, and before Rhythm and Hues permanently closed its doors, Neulander took his computer graphics skills to a software engineering post at Google, where he started on the photos team and worked on painterly renderings.

“When I first started, VR [virtual reality] was still in its infancy. The technology wasn’t yet there, to make a truly compelling experience,” says Neulander, who works out of the famous Binoculars Building, an office space designed by Canadian-born architect Frank Gehry in Venice, Calif.

“I was interested primarily in computer graphics, and more specifically, photorealistic rendering. And so it lent itself really well to visual effects, to movies. And that's the industry that I pursued.

“It wasn't until 15 or so years later, when I came to Google, that the Daydream team was gathering steam and we had the tech to render things smoothly in a way that doesn't give people nausea. We could actually start using mobile devices to give everybody a pretty good VR experience.”

Neulander explains rendering as image synthesis – the creation of an image on a computer.

“The beauty of computer graphics is…the image. I like the fact that I can kind of validate what I've done just by looking at the image.”

In detail, rendering is 3D and requires a virtual camera, virtual light sources, 3D geometry (in the form of triangular meshes, textures that are applied to the surface), followed by shading tricks to ensure the surfaces look realistic, whether it's fibres, or wood, or even clouds, he says.

“It can be something that happens 60 times per second on a PlayStation 4, which gives you actually a pretty decent level of quality these days. But when you're talking about visual effects, there's a much higher bar for the quality that's expected. The shadows have to look really good. The lighting has to look realistic. Oftentimes you're inserting computer-generated content into live action footage – people interacting with CG [computer graphics] characters. And so the lighting really has to match between those two.”

Lighting a scene with "blobby" shadows or shadow "maps" are some of the problems Neulander tackled for Google’s development of AR (augmented reality) stickers, a camera mode currently available in Google’s Pixel phones, allowing users to place CG characters – including some from Star Wars, Stranger Things, among Google’s own – in the phone’s live view. It’s the newest app to be developed using Google’s ARCore technology.

Disney’s Star Wars “porg” visits the Department of Computer Science Innovation Lab via the Google AR stickers app (photo by Ryan Perez)

“I feel this is kind of a new development with mobile phones,” he says. “AR stickers are meant to be fun and kind of lighthearted. It's a neat way to get the [AR] ball rolling.”

Neulander predicts future uses could help you navigate a store, or place notes on places you want to revisit.

Could we one day augment our view by looking at buildings through our phones and gather their history?

“Being able to quickly identify buildings based on what the phone sees, is exactly the kind of problem that Google excels at. It's something that lends itself well to a deep learning solution, and there's so much training data that Google has, to feed that learning algorithm.

“Stay tuned, but I think something like that will be coming very soon.”

Interested in AI and augmented reality? ModiFace creates 50 internships for U of T engineering undergrads

ullahnor — Tue, 11 Apr 2017 16:03:39 +0000

Interested in AI and augmented reality? ModiFace creates 50 internships for U of T engineering undergrads

ullahnor Tue, 04/11/2017 - 12:03

Cutline

ModiFace technology lets users virtually manipulate their appearance, trying on makeup products and hair styles in 3D and real time. The company is hiring 50 interns (photo courtesy of ModiFace)

Marit Mitchell

Author legacy

Marit Mitchell

Topic

Breaking Research

Artificial Intelligence

Augmented Reality

Faculty of Engineering & Applied Science

Students

Undergraduate Students

Internships

Subheadline

Company founded by U of T engineering associate professor gives $4 million to support undergraduate, graduate internships and research at Canada’s top-ranked engineering school

Augmented reality startup ModiFace is investing $4 million to create new undergraduate and graduate student internships, and support leading research at U of T's Faculty of Applied Science & Engineering.

The company, founded by Associate Professor Parham Aarabi of the department of electrical & computer engineering, uses augmented reality (AR) and artificial intelligence (AI) to build advanced facial visualization software for the beauty and medical industries. ModiFace technology powers over 100 AR applications by Fortune 500 brands including Sephora, L’Oreal, Allergan, Vichy and Clairol, among others.

“The future of ModiFace is highly dependent on our access to the best AR engineers in the world,” says Aarabi. “For AR, it takes about a year for a new graduate to get up to speed with the latest concepts in artificial intelligence, systems engineering and computer vision. As a result, we want to invest in students early while they’re still in school to start giving them the best training in these fast-moving fields.”

As part of U of T Engineering’s Professional Experience Year (PEY) internship program, ModiFace is hiring 50 undergraduate students into 12- to 16-month paid positions that allow students to gain extensive professional skills before graduation. Applications are now open, and the first cohort will start in May 2017.

The company is also launching ModiFace research internships, modelled after successful research programs by top tech companies in the U.S., which will allow graduate students to complete an enriched research internship during their master’s or PhD studies. A total of 10 graduate students will be undertaking these internships in the next two years. Simultaneously, ModiFace is establishing a research grant program to support industry-partnered research in fields from artificial intelligence and deep learning, to computer vision and graphics.

“ModiFace perfectly illustrates why U of T is the top university for research-based startups in North America,” says Cristina Amon, dean of the Faculty of Applied Science & Engineering. “This investment in enriched experiential learning opportunities for our students further reinforces the strong relationship between the excellence of our innovative engineering programs and the thriving entrepreneurial ecosystem around us.”

“ModiFace’s internship programs will help Ontario’s engineering students get the skills and training they need to succeed in the growing field of augmented reality,” said Reza Moridi, Ontario minister of research, innovation and science. “The initiative dovetails with our government’s innovation strategy, including the recent launch of Ontario’s Vector Institute for Artificial Intelligence, which will help Ontario continue to attract top talent and strengthen our province’s innovation and knowledge-based economy.”

The announcement builds on the recent creation of the Vector Institute, a multidisciplinary research and commercialization hub that aims to solidify Toronto’s place as a global AI destination.

Taking video games to the next level: U of T’s Steve Engels on this year’s student game design showcase

ullahnor — Mon, 03 Apr 2017 18:57:59 +0000

Taking video games to the next level: U of T’s Steve Engels on this year’s student game design showcase

ullahnor Mon, 04/03/2017 - 14:57

Cutline

Level Up brings together game design students and gaming enthusiasts from all over Ontario (photo by Nina Haikara)

Nina Haikara

Author legacy

Nina Haikara

Topic

Over 2,000 students, instructors, industry insiders and gaming fans will crowd the Design Exchange for the seventh annual Level Up Showcase on Wednesday, an event that has become known for uncovering new creative talent.

Steve Engels, an associate professor, teaching stream in U of T's department of computer science, co-founded the event with Emma Westecott, an assistant professor at OCAD University, seven years ago.

“This event brings together game design students and gaming enthusiasts from all over Ontario, showcasing Ontario’s young talent to industry, media and the world,” says Engels.

“To see this event grow from its humble beginnings to what it is today is truly inspiring. It really shows how the field continues to grow and expand into all sorts of exciting new directions, and we’re looking forward to seeing all the things our students come up with this year.”

Pokémon GO craze shows that augmented reality is hitting its stride, U of T experts say

lavende4 — Wed, 13 Jul 2016 13:39:01 +0000

Pokémon GO craze shows that augmented reality is hitting its stride, U of T experts say

lavende4 Wed, 07/13/2016 - 09:39

Cutline

Playing Pokémon GO outside the Nintendo flagship store in New York (Drew Angerer/Getty Images)

Terry Lavender

Author legacy

Terry Lavender

Topic

Faculty of Applied Science & Engineering

Faculty of Art & Science

Culture

Subheadline

“A perfect combination of an application that combines nostalgic characters, easy graphics, a relatively low need for accuracy, mobility, and community”

It's not even available in Canada yet, but Pokémon GO, the new augmented reality (AR) app from Niantic Labs, is dominating watercooler conversations, social media and headlines both here and abroad.

The GPS-enabled game for Android and Apple smartphones is based on the Nintendo Pokémon characters. Players wander around searching for Pokémon, which the game triggers at semi-random physical locations. When they find one, it activates the phone’s camera and shows them the Pokémon.

While Pokémon Go is a new phenomenon, U of T researchers have been working with augmented reality for years. U of T News spoke to computer science professor Eugene Fiume and electrical and computer engineering professor Steve Mann about Pokémon GO and the future of augmented reality.

Eugene Fiume

What is augmented reality (AR)?
As the name would suggest, augmented reality (AR) is a computer-aided way to augment the physical world by inserting computer generated information into the view of a real scene. Normally, this is done by streaming one channel or more of video of the world into a device and then inserting computer synthesized imagery into those channels just in time for interactive viewing. For the most part, augmented reality is about continuous visual phenomena, though one could imagine augmenting sound, images, touch, and one day perhaps even taste. Common uses for AR include museum exhibits, automatic sign translation, interactive games, mapping, defect analysis, artistic tools, see-through displays on cars, etc. AR should be distinguished from virtual reality (VR), whose purpose is to provide an immersive synthetic approximation of reality whose use generally is isolated from "real" reality. AR blends the real and the virtual, while VR replaces the real by the virtual. These distinctions will in time blur.

How does it work?
The main technology needed for AR is the ability to recognise some aspects of how a user is moving and the position and direction of the camera. If I know where a person is and how he or she is holding the camera, I can then approximate (with a little modelling) apparent features in the real scene, such as walls, roads, and posts. This kind of information comes from accelerometers and position sensors: the same things that our fitness applications use to track our distance and speed. As well, some image processing, computer vision and AR is often used to recognise objects or just basic geometry such as a large flat region in the incoming video stream. Then computer graphics jumps in to generate synthetic images (signs, Pokémon characters, narratives, animation) and blends them into the video stream just before it appears on the viewer's screen.

Augmented reality apps have been around for several years now. Why has Pokémon GO suddenly become so popular?
AR (and its cousin, virtual reality) have been around for a long time. Steve Feiner from Columbia University did AR walking around with a huge backpack, doing a lot of the stuff that is becoming commonplace on a handset today. But AR is only now hitting its stride. Generally as a technology this has to do with Moore's Law combining with devices such as mobile handsets that integrate sensors and functions such as cameras, GPS, accelerometers, image processing, graphics, etc. So now we're starting to see AR become commonplace. But the problem is that the fundamental functionality that AR needs such as registration of live video to a synthetic character or sign is quite hard to accomplish for austere applications that require accuracy. And now we come to Pokémon GO. Here we have a perfect combination of an application that combines nostalgic characters, easy graphics, a relatively low need for accuracy, mobility, and community.

Will its popularity be good for augmented reality apps, or will it lead developers to focus on apps with a narrow commercial potential?
I think its popularity will be good for AR overall, as long as people understand that AR has a very broad scope, and that there are many unsolved problems with this technology (as well as with Virtual Reality).

What other AR apps would you recommend to people who’ve become interested because of Pokémon GO?
There are some terrific little applications that do things like sign translation (Google Translate), Crayola Colo[u]r, Wikitude. There are some beautiful star-chart semi-AR applications that allow you to explore the night sky.

Pokémon GO is an entertainment app. Can you give examples of serious uses for augmented reality?
AR is already part of the landscape in high-tech situations, for example, BMW's Mini augmented glasses, which you can see here. I particularly like "austere" AR applications that are being developed now to do defect analysis in cars or boats, or to do 3D/AR training. See, for example, www.ngrain.com.

What is happening at U of T in terms of AR research?
Karan Singh, in our Dynamic Graphics Project lab, is doing work on VR and has a startup called JanusVR. Steve Mann, in Electrical and Computer Engineering, has been doing various forms of ground-breaking AR for many years.

A lot of the media coverage of Pokémon GO is about dangerous consequences – players hurting themselves because they’re not paying attention; players being lured to remote locations where they’ve been robbed, etc. But does Augmented Reality pose any dangers?
Looking at the world through a narrow portal is potentially dangerous, but certainly AR is not the only possible concern. The problem here is one of attentional focus as well as "change blindness". The former reflects ongoing research that humans are in fact terrible multi-taskers. The latter points to the fact that in many situations we do not detect changes in our visual field. There are some very graphic examples of this, but the nicest one I've seen is a commercial (!) for ŠKODA. In fact, AR systems could well be better "augmented" to detect environmental changes.

Steve Mann

You are considered one of the pioneers of augmented reality. Tell us about your involvement with AR.
I've been doing mobile, portable, and wearable AR for 42 years now, as a way to help people see and understand their world. Games often accelerate technologies that have been around for a long time, but technologies of the present day make it easier to do. In my original system, a lot of heavy radio and wireless communications equipment was necessary, but today these features are built into smartphones. Some of the tracking systems I developed many years ago that required a large computer can now be implemented in a small device like a smartphone.

Computer games were responsible for accelerated computer development, such as GPGPU (our lab was the first to use parallel GPGPU systems for general-purpose computation, even though they were designed mainly for computer graphics processing as needed in video games).

Will the success of Pokémon GO be good for AR?
Games like Pokémon GO help to make more people aware of the concepts of Augmented and Mediated ("Augmediated") Reality, and the fact that these realities not only augment our vision but modify (mediate) it.

Is Pokémon GO a good example of the current state of AR development?
The current state of AR development goes way beyond what's implemented in these computer games. Take for example the Metavision glasses that implement a Natural User Interface (NUI). I coined the term "NUI" back in the 1990s and built a natural computational system at MIT Media Lab founding the MIT Wearable Computing project as its first member.

Pokémon GO is a smartphone app. Are smartphones the future of AR? What about other platforms, such as eyewear or other wearables?
Smartphones are not the future of AR, but merely a transitional technology on the way to something I called "Digital Eye Glass" originally. DEG (Digital Eye Glass) like EyeTap or Metavision represents the future of AR.

Augmented Reality

U of T and Autodesk researchers design 2D-3D software for augmented reality drawing

French cosmetics giant buys U of T 'beauty tech' startup

Read about the deal at CNN

Read about the deal at the Globe and Mail

Learn more about U of T Entrepreneurship

What's it like to develop Google’s latest AR technology? Ask this U of T alumnus

Interested in AI and augmented reality? ModiFace creates 50 internships for U of T engineering undergrads

Read more about the Vector Institute

Taking video games to the next level: U of T’s Steve Engels on this year’s student game design showcase

Read more about Level Up Showcase

Pokémon GO craze shows that augmented reality is hitting its stride, U of T experts say

Eugene Fiume

Steve Mann