Augmenting Humans and AI with User Interfaces

XIE Laboratory
Associate Professor：XIE Haoran

E-mail：
［Research areas］
Computer Graphics, User Interface, Human Augmentation
［Keywords］
Deep Learning, Sketch, Human-AI Interaction, Human-Centered AI, Physical Augmentation, Wearable Device

Skills and background we are looking for in prospective students

Strong desire to conduct the challenging topics in novel interdisciplinary research fields and realize their own "dreams". It is desirable but not required, willing to master the graphics and machine learning technologies.

What you can expect to learn in this laboratory

Students can develop future value-creating capabilities with professional expertise and wide knowledge in the coming super-smart society. They can gain the ability to think cross-disciplinarily, utilize feasible approaches, and create innovations. We encourage students to accumulate knowledge from different fields of study and foster interdisciplinary thinking. The development of augmented interfaces acquires practical skills for actual social issues.

【Job category of graduates】
ICT, R&D, Manufacturing, Creative Industry, etc.

Research outline

We aim to conduct the advanced topics in Augmented(X) Interfaces for realizing Society 5.0^※ (Fig. 1). We are proposing user interfaces that enable even novice users to democratize the advanced artificial intelligence and robotics technologies which are only available only by domain experts. We aim to realize a future society in which anyone can pursue own dreams by enabling users to augment own creativity, physical capabilities, and cognitive abilities. Our research vision is to realize the super human and super information technology through the interface technology in the creative society, aiming at the social implementation that can harmonize the human and machine.
※super-smart society, the concept of the future society proposed in Japan 5th Science and Technology Basic Plan.

(1) Sketch Interfaces with Creative AI

The rapid development of deep learning related technologies has led to social reforms nowadays. However, creative artificial intelligence (AI) technology is still a challenging and important research topic attracting worldwide attention. This research topic aims to support the media content production by using data-driven methods such as deep learning based generative models. Specific examples include design interface that supports the interactive design of 3D airplane geometry and generative interface that enables to edit the flow motion from user sketches.

(2) Augmented AI with Human Impact

This research topic aims to conduct the user interfaces to consider human impact in the loop of machine learning process including interfaces for human-AI interaction, human-in-the-loop machine learning and human computation. Specific examples include annotation interface for data collection using crowdsourcing, and 3D model construction from point clouds.

(3) Augmented Human with AI Technologies

Recent developments of AI, XR and 5G technologies are augmenting every aspect of our lives. We are in the "Augmented Age" which offers us the greatest advantages and possibilities in human history. This research topic aims to develop technologies to recover or augment human's capabilities using information technology and robotics. Specific examples include wearable device that mimics the animal tails to augment human body functions and emotional expressiveness, and the wearable supernumerary robotic limbs to support humans during everyday activities.

Key publications

1. Z. Huang, Y. Peng, T. Hibino, C. Zhao, H. Xie, T. Fukusato, K. Miyata. dualFace: Two-Stage Drawing Guidance for Freehand Portrait Sketching. Journal of Computational Visual Media, Vol 8, 63-77, 2022.
2. Z. Hu, H. Xie, T. Fukusato, T. Sato and T. Igarashi. Sketch2VF: Sketch-Based Flow Design with Conditional Generative Adversarial Network. Computer Animation and Virtual Worlds, 30(3-4),1-11, 2019.
3. H. Xie, T. Igarashi and K. Miyata. Precomputed Panel Solver for Aerodynamics Simulation. ACM Transactions on Graphics. 37(2), 1-12, 2018. (ACM SIGGRAPH 2018)

Equipment

Deep Learning Server, 3D Printer and Scanner
Sensors, Projectors, Depth Cameras, JAIST FabLab