In the adenoma-carcinoma sequence, it is proposed that intestinal polyps evolve through a set of defined mutations toward metastatic colorectal cancer (CRC). Here, we dissect this adenoma-carcinoma sequence in vivo by using an orthotopic organoid transplantation model of human colon organoids engineered to harbor different CRC mutation combinations. We demonstrate that sequential accumulation of oncogenic mutations in Wnt, EGFR, P53, and TGF-β signaling pathways facilitates efficient tumor growth, migration, and metastatic colonization. We show that reconstitution of specific niche signals can restore metastatic growth potential of tumor cells lacking one of the oncogenic mutations. Our findings imply that the ability to metastasize—i.e., to colonize distant sites—is the direct consequence of the loss of dependency on specific niche signals.