Reinforcement learning has made significant progress in solving sequential decision problems under uncertainty. However, reinforcement learning agents generally lack a fundamental understanding of the world and must therefore learn from scratch through numeroustrial-anderror interactions. They may also face challenges in providing explanations for their decisions and generalizing the acquired knowledge. Causality presents a promising approach to address these issues by formalizing knowledge in a systematic manner and leveraging invariance for effective knowledge transfer. This tutorial aims to comprehensively review the emerging field of causal reinforcement learning. We will introduce the basic concepts of causality and reinforcement learning and demonstrate how causality can enhance traditional reinforcement learning algorithms. The tutorial will categorize and systematically review existing causal reinforcement learning approaches based on their target problems and methodologies. We will also outline open issues and future research directions to foster the continuous development and application of causal reinforcement learning in real-world scenarios. We believe that this tutorial will contribute significantly to the data mining community, offering a unique perspective on integrating causality into reinforcement learning and providing participants with valuable knowledge to explore this emerging field.

The tutorial will consist of two informative parts, followed by a Q&A session.

Part 1: Background on Reinforcement Learning and Causality
1. A high-level introduction to causal reinforcement learning.
2. Overview of causality: mathematical formulation and fundamental concepts.
3. Overview of reinforcement learning: definitions, fundamental concepts and categories.
4. Critical challenges in non-causal reinforcement learning.
Part 2: Advancements in Causal Reinforcement Learning
5. Enhancing sample efficiency through causal reinforcement learning.
6. Advancing generalizability and knowledge transfer through causal reinforcement learning.
7. Addressing spurious correlations through causal reinforcement learning.
8. Considerations beyond return.
9. Open problems and future directions.

Jing Jiang is an Associate Professor in the School of Computer Science, a core member of Australian Artificial Intelligence Institute (AAII), at the University of Technology Sydney (UTS) in Australia. Her research interests focus on machine learning and its applications. She is the recipient of the DECRA(Discovery Early Career ResearcherAward)fellowship funded by ARC (Australian Research Council). She has published over 70 papers in the related areas of AI in the top-tier conferences and journals, such as NeurIPS, ICML, ICLR, AAAI, IJCAI, KDD, TNNLS and TKDE.

Zhihong Deng received his undergraduate and master degree in computer science in 2017 and 2020 respectively from Sun Yat-sen University. He is pursuing a PhD degree at University of Technology Sydney. His research interests span machine learning and data mining, with a special focus on reinforcement learning. He has published papers in multiple international conferences and journals, such as AAAI, ICLR, IEEE TCYB and IEEE TNNLS.

The exponential growth of digital data requires the development of efficient machine learning models. Nonetheless, it is ubiquitous that data in different scenarios (domains) conform to different underlying distributions. A machine learning model that has been trained in a specific domain may not work well when presented with data from other domains. Domain adaptation arises when we aim to exploit heterogeneous information from different but related domains to transfer knowledge to the target domain of interest. With the aid of domain adaptation technology, one can build more effective machine learning models that can perform well in a wide range of real-world scenarios.

This tutorial illustrates the techniques and strategies used in domain adaptation, allowing one to build more accurate and reliable machine learning models that can adapt to different types of data. With clear explanations and experimental examples, this tutorial is essential for anyone who hopes to deepen their understanding of this important subarea in transfer learning.

Domain adaptation is a crucial technique in the field of machine learning that allows models to adapt to different types of data. With the increasing availability of data from diverse sources, there is a growing demand for domain adaptation techniques in a wide range of industries and applications, making it a valuable skill set for researchers, students, and industry professionals, etc. This tutorial systemically describes the key contents and covers the recent advancements of domain adaptation, including conventional unsupervised domain adaptation and other promising variants. It would be valuable for researchers, students, and industry professionals to keep up-to-date on the latest advancements in this important area of machine learning.

Jingjing Li received the M.Sc. and Ph.D. degrees in computer science from the University of Electronic Science and Technology of China in 2013 and 2017, respectively. He is currently a full Professor with the School of Computer Science and Engineering, University of Electronic Science and Technology of China. His current research interests include computer vision, machine learning and multimedia analysis, especially transfer learning, domain adaptation and zero-shot learning. Dr. Jingjing Li has published over 70 peer-reviewed papers on top-ranking journals and conferences, including IEEE TPAMI, TIP, TKDE, CVPR and NeurIPS. He has long served as a reviewer/PC/SPC/AC for TPAMI, TIP, TOIS, AAAI, CVPR, WACV and ACM MM. He won the Excellent Doctoral Thesis Award of The Chinese Institute of Electronics, and The Excellent Young Scholar of Wu Wen Jun AI Science & Technology Award.

Heterogeneous graphs, also known as heterogeneous networks or heterogeneous information networks, are composed of different types of entities and relationships. Heterogeneous graphs are a fundamental tool for modeling complex interactive systems and a widely existing form of data, forming unique analytical methods. Heterogeneous graph computing has also become a research hotspot in the field of data mining and is widely used in the industry. This report systematically introduces the basic concepts of heterogeneous graphs, meta path based analysis methods, shallow and deep representation models, as well as applications and platforms.

Chuan Shi is the professor and PhD supervisor in School of Computer Sciences of Beijing University of Posts and Telecommunications, deputy director of Beijing Key Lab of Intelligent Telecommunication Software and Multimedia. The main research interests include data mining, machine learning, artificial intelligence and big data analysis. He has published more than 100 refereed papers, including top journals and conferences in data mining and machine learning, such as IEEE TKDE, KDD, WWW, ICLR, NeurIPS, AAAI, IJCAI and ACL. He has been honored as the best paper award in ADMA 2011/ADMA 2018 and the best paper nomination in the WebConf 2021. He also won several awards, such as the first prize of science and technology progress of the Chinese Institute of Electronics (1st) and the second prize of Natural Science of Beijing/CCF (1st).