Abstract: It’s commonly difficult for autonomous vehicles (AVs) to detect human driver distraction behaviors, which would lead to probable late preparation for AVs to take brakes in time to avoid rear-end collisions. Hence, in this paper, we aimed to build a connection between AV control and human driver distraction detection, to aid AVs to predict potential risk and avoid distracted drivers like experienced human drivers. First, a practical system framework integrating driver distraction detection, vehicle-to-vehicle communication, and AV velocity control was proposed. Then, an implementation of driver distraction detection based on convolutional neural networks was put forward. Finally, a longitudinal velocity control method considering driver distraction behavior based on model predictive control was posed with persistent feasibility analysis. Simulation results validated the effectiveness of the work in this paper.