Research

The long-term research goal is to build understand, evaluate, and enhance modern AI models, such as pre-trained and large foundation models. We create new theory, algorithms, applications, and open-sourced library to achieve our goal. These days, we are specifically interested in large language models (LLMs) evaluation and robustness enhancement.

Our research consists of the following topics with selected publications: [View by year] [Google scholar]

New: large models

Evaluation: (website: https://llm-eval.github.io/)

• [arXiv’24] DyVal 2: Dynamic Evaluation of Large Language Models by Meta Probing Agents. Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, Xing Xie.
• [ICLR’24 spotlight] DyVal: Graph-informed Dynamic Evaluation of Large Language Models. Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, Xing Xie.
• [ICLR’24] PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, Yue Zhang. [code]
• [arXiv’23] CompeteAI: Understanding the Competition Behaviors in Large Language Model-based Agents. Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, Xing Xie.
• [arXiv’23] Meta Semantic Template for Evaluation of Large Language Models. Yachuan Liu, Liang Chen, Jindong Wang, Qiaozhu Mei, Xing Xie.
• [arXiv’23] A Survey on Evaluation of Large Language Models. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, Xing Xie. [code]
• [arXiv’23] PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, Xing Xie. [code]
• [ACL’23 findings] GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective. Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, Yue Zhang.
• [ICLR’23 large model workshop] On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective. Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang, and Xing Xie.

Enhancement: (website: https://llm-enhance.github.io/)

• [arXiv’24] On Catastrophic Inheritance of Large Foundation Models. Hao Chen, Bhiksha Raj, Xing Xie, Jindong Wang.
• [arXiv’24] CultureLLM: Incorporating Cultural Differences into Large Language Models. Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, Xing Xie.
• [ICLR’24 spotlight] Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks. Hao Chen, Jindong Wang, Ankit Shah, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, Bhiksha Raj.
• [ICLR’24] Supervised Knowledge Makes Large Language Models Better In-context Learners. Linyi Yang, Shuibai Zhang, Zhuohao Yu, Guangsheng Bao, Yidong Wang, Jindong Wang, Ruochen Xu, Wei Ye, Xing Xie, Weizhu Chen, Yue Zhang.
• [arXiv’23] ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning. Wang Lu, Hao Yu, Jindong Wang, Damien Teney, Haohan Wang, Yiqiang Chen, Qiang Yang, Xing Xie, Xiangyang Ji.
• [arXiv’23] EmotionPrompt: Leveraging Psychology for Large Language Models Enhancement via Emotional Stimulus. Cheng Li, Jindong Wang, Kaijie Zhu, Yixuan Zhang, Wenxin Hou, Jianxun Lian, Xing Xie.
• [IJCV’23] Exploring Vision-Language Models for Imbalanced Learning. Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, Shikun Zhang. [code]
• [IEEE Data Engineering Bulletin’23] FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning. Wang Lu, Xixu Hu, Jindong Wang, Xing Xie. [arxiv]

Open source:

• Project SearchAnything: semantic local search.

Out-of-distribution (Domain) generalization and adaptation for distribution shift

• [IEEE TPAMI’24] DIVERSIFY: A General Framework for Time Series Out-of-distribution Detection and Generalization. Wang Lu, Jindong Wang, Xinwei Sun, Yiqiang Chen, Xiangyang Ji, Qiang Yang, Xing Xie.
• [IEEE VR 24] Generating Virtual Reality Interaction Data from Out-of-Distribution Desktop Data: An Exploration Using Stroke Gestures. Linping Yuan, Boyu Li, Jindong Wang, Huaming Qu, Wei Zeng.
• [UbiComp’24] Optimization-Free Test-Time Adaptation for Cross-Person Activity Recognition. Shuoyuan Wang, Jindong Wang, HuaJun Xi, Bob Zhang, Lei Zhang, Hongxin Wei.
• [CPAL’24] Optimization-Free Test-Time Adaptation for Cross-Person Activity Recognition. Shuoyuan Wang, Jindong Wang , Huajun Xi, Bob Zhang, Lei Zhang, and Hongxin Wei.
• [TNNLS’24] UP-Net: An Uncertainty-Driven Prototypical Network for Few-Shot Fault Diagnosis. Ge Yu, Jindong Wang, Jinhai Liu, Xi Zhang, Yiqiang Chen, and Xing Xie.
• [NeurIPS’23] Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models. Andy Zhou, Jindong Wang, Yu-Xiong Wang, Haohan Wang.
• [ICCV’23] Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning. Kaijie Zhu, Xixu Hu, Jindong Wang, Xing Xie, Ge Yang.
• [ICLR’23] Out-of-distribution Representation Learning for Time Series Classification. Wang Lu, Jindong Wang, Xinwei Sun, Yiqiang Chen, and Xing Xie.
• [KDD’23] Domain-Specific Risk Minimization for Out-of-Distribution Generalization. YiFan Zhang, Jindong Wang, Jian Liang, Zhang Zhang, Baosheng Yu, Liang Wang, Xing Xie, and Dacheng Tao.
• [KDD’23] Generalizable Low-Resource Activity Recognition with Diverse and Discriminative Representation Learning. Xin Qin, Jindong Wang, Shuo Ma, Wang Lu, Yongchun Zhu, Xing Xie, Yiqiang Chen.
• [ACL’23 findings] GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective. Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, Yue Zhang.
• [KDD’23 workshop] Towards Optimization and Model Selection for Domain Generalization: A Mixup-guided Solution. Wang Lu, Jindong Wang, Yidong Wang, Kan Ren, Yiqiang Chen, Xing Xie.
• [TKDE’22] Generalizing to Unseen Domains: A Survey on Domain Generalization. Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip S. Yu.
• [TMLR’22] Domain-invariant Feature Exploration for Domain Generalization. Wang Lu, Jindong Wang, Haoliang Li, Yiqiang Chen, and Xing Xie.
• [UbiComp’22] Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition. Wang Lu, Jindong Wang, Yiqiang Chen, Sinno Pan, Chunyu Hu, and Xin Qin.
• [NeurIPS’21] Learning causal semantic representation for out-of-distribution prediction. Chang Liu, Xinwei Sun, Jindong Wang , Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie-Yan Liu.
• [CIKM’21] Adarnn: Adaptive learning and forecasting of time series. Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang.
• [TNNLS’20, 400 + citations] Deep subdomain adaptation network for image classification. Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke, Jingwu Chen, Jiang Bian, Hui Xiong, and Qing He.
• [ACMMM’18, 500+ citations] Visual domain adaptation with manifold embedded distribution alignment. Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu.
• [ICDM’17, 500+ citations] Balanced distribution adaptation for transfer learning. Jindong Wang, Yiqiang Chen, Shuji Hao, Wenjie Feng, and Zhiqi Shen.
• Open-source:
  • Transfer learning
  • robustlearn: A unified repo for robust machine learning, such as OOD and adversarial robustness: robustlearn
  • PandaLM: PandaLM

Semi-supervised learning for low-resource learning

• [ICLR’23] FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning. Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie.
• [ICLR’23] SoftMatch: Addressing the Quantity-Quality Tradeoff in Semi-supervised Learning. Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides.
• [NeurIPS’22] USB: A Unified Semi-supervised Learning Benchmark. Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj, Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang.
• [TASLP’22] Exploiting Adapters for Cross-lingual Low-resource Speech Recognition. Wenxin Hou, Han Zhu, Yidong Wang, Jindong Wang, Tao Qin, Renjun Xu, and Takahiro Shinozaki.
• [NeurIPS’21, 200+ citations] Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki.
• Open-source:
  • USB: A unified semi-supervised learning toolbox for CV, NLP, and Audio: USB
  • A unified Pytorch-based semi-supervised learning library: **TorchSSL

Safe transfer learning for security

• [arXiv’24] SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization. Xixu Hu, Runkai Zheng, Jindong Wang, Cheuk Hang Leung, Qi Wu, Xing Xie.
• [TNNLS’23] MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcare. Yiqiang Chen, Wang Lu, Xin Qin, Jindong Wang , and Xing Xie.
• [ICSE’22] ReMoS: Reducing Defect Inheritance in Transfer Learning via Relevant Model Slicing. Ziqi Zhang, Yuanchun Li, Jindong Wang, Bingyan Liu, Ding Li, Xiangqun Chen, Yao Guo, and Yunxin Liu.
• [IEEE Data Engineering Bulletin’23] FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning. Wang Lu, Xixu Hu, Jindong Wang, Xing Xie. [arxiv]
• [IEEE TBD’22] Personalized Federated Learning with Adaptive Batchnorm for Healthcare. Wang Lu, Jindong Wang, Yiqiang Chen, Xin Qin, Renjun Xu, Dimitrios Dimitriadis, and Tao Qin.
• [TKDE’22] Unsupervised deep anomaly detection for multi-sensor time-series signals. Yuxin Zhang, Yiqiang Chen, Jindong Wang, and Zhiwen Pan.
• [IntSys’22, 400+ citations] Fedhealth: A federated transfer learning framework for wearable healthcare. Yiqiang Chen, Xin Qin, Jindong Wang, Chaohui Yu, and Wen Gao.
• Open-source:
  • PersonalizedFL: a personalized federated learning libraty: PersonalizedFL
  • robustlearn: A unified repo for robust machine learning, such as OOD and adversarial robustness: robustlearn

Imbalanced learning for long-tailed tasks

• [arXiv’23] Exploring Vision-Language Models for Imbalanced Learning. Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, Shikun Zhang.
• [ACML’22] Margin Calibration for Long-Tailed Visual Recognition. Yidong Wang, Bowen Zhang, Wenxin Hou, Zhen Wu, Jindong Wang, and Takahiro Shinozaki.
• Open-source:
  • Imbalance-VLM: a library for imbalanced learning in vision-language models. [Imbalance-VLM]

Miscellaneous

1. An easy-to-use speech recognition toolkit based on Espnet: EasyESPNet
2. Leading the transfer learning tutorial (迁移学习简明手册) on Github: Tutorial
3. I’m also leading other popular research projects: Machine learning, Activity recognition
4. I started a software studio Pivot Studio and made many applications in 2010-2014: Our applications