Cross-lingual embeddings (CLE) allow for cross-lingual natural language processing and information retrieval. Recently, a wide variety of resource-lean projection-based models for inducing CLEs appeared, requiring limited or no bilingual supervision. Despite potential usefulness in downstream IR and NLP tasks, these CLE models have almost exclusively been evaluated on word translation tasks. In this work, we provide a comprehensive comparative evaluation of projection-based CLE models for both sentence-level and document-level Cross-lingual Information Retrieval (CLIR). We hope our work serves as a guideline for choosing the right model for CLIR practitioners.