The complexity of AI-empowered autonomous systems introduces great design challenges, which require multimodal multi-task (MMMT) learning while being aware of hardware performance and implementation strategies. However, the MMMT learning in autonomous systems is still underexplored. In this paper, we first discuss the opportunities of applying MMMT in autonomous systems and the unique challenges. We then discuss the necessity and opportunities of the MMMT model and hardware co-design, which is critical for autonomous systems, especially with heterogeneous platforms. We formulate the co-design as a differentiable optimization problem and advocate for further explorations of MMMT in autonomous systems and software/hardware co-design solutions.