Overview
To facilitate the study of indoor scene building methods and their potential
applications for robotics, animation, and Embodied AI, we introduce VRKITCHEN2.0: a toolkit built by NVIDIA OMNIVERSE that provides flexible pipelines for indoor scene building, scene randomizing, and robotic controls.
Besides, by combining Python coding in the animation software VRKITCHEN2.0 can assist researches in creating real-time training and control for robotics in the future.
Demo 1: Build customized high-quality indoor scenes
Embodied artificial intelligence (EAI) has attracted significant attention, both in advanced deep learning models and algorithms and the rapid development of simulated platforms. Many open challenges have been proposed to facilitate EAI research. A critical bottleneck in existing simulated platforms is the limited number of indoor scenes that support vision-and-language navigation, object interaction, and complex household tasks [ZLJ+21].
In this tutoral: Tutorial x: Indoor scene bulding, we shall present how to import tens of thousands of room layouts from the original 3D-Front dataset into Omniverse to give a photo-realistic effect for EAI tasks.
Demo 2: Parsing articulated objects
Articulated objects can be defined as objects composed of more than one rigid parts. In our daily life, humans are constantly interacting with a lot of articulated objects such as door, keyboard, light switch, and e.t.c. The rigidbody, softbody, articulated object, and liquid compose a large part of our interaction with the world.
In this tutorial: Tutorial x: Parsing Articulated Object, we present how to parse articulated objects in SAPIEN [XQM+20] (a realistic and physics-rich simulated environment) into Omniverse, and present their potential applications for dynmaic controls in the virtual environment.
Demo 3: Set up liquid
Research on studies of softbodies and liquid is very prolific and already achieved important success, although with very different technological appliciable level. The interaction with liquid and softbody in Embodied AI is by far the most productive topic.
In this tutorial: Tutorial x: Set up liquid, we show how to calculate and build liquid in Omniverse.
Demo 4: Set up Robot
Embodied AI originally refers to AI for virtual robots, which is the field for solving AI problems for virtual robots that can move, see, speak, and interact in the virtual world and with other virtual robots/ Hopefully, the simulated robot solutions are then transferred to real world robots.
In this tutorial (Tutorial x: Set up a Franka-panda robot), we show how to import a Franka Emika Panda robot in Omniverse.
Demo 5: (Put together) Data labeling
Machine Learning and Deep Learning have been successful because of the increasing amounts of data and the increasing amounts of computing power (e.g., CPUs, GPUs, TPUs). However, this type of “trainable data” for Embodied AI and robotics are still limited especially when we consider them in virtual environment.
To enrich the dataset for robotics and EAI community, this tutorial (Tutorial x: Data labeling with scenes, articulated objects, liquid and robots) teaches how to integrete everything together to label a robot task in a real scene with real physics with scenes, articulated objects, liquid and robots.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, and others. Sapien: a simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11097–11107. 2020.
Yizhou Zhao, Kaixiang Lin, Zhiwei Jia, Qiaozi Gao, Govind Thattai, Jesse Thomason, and Gaurav S Sukhatme. Luminous: indoor scene generation for embodied ai challenges. arXiv preprint arXiv:2111.05527, 2021.