During membrane trafficking between the trans Golgi network (TGN) and endosomes, clathrin-coated vesicles are formed through the concerted action of adaptor proteins that link clathrin, membrane and cargo. Here, Daboussi et al. use live cell imaging and super-resolution microscopy to dissect the relationship between the adaptor proteins Ap1 and Gga2 in budding yeast. They find that these proteins are recruited to the TGN in two waves during coat assembly, with Gga2 arriving first. Levels of the phosphoinositide PtdIns4P, which mediates the localization of these adaptors to the TGN, are important for ensuring the proper rate of adaptor progression. In addition, Gga2 recruits Pik1, a kinase that generates PtdIns4P, through a direct interaction. Thus, this mechanism may provide a means of generating adaptor-specific coats and coated vesicles with distinct target destinations.