Accurate modeling of protein-ligand interactions (PLIs) is crucial for drug discovery. Despite progress, most existing PLI modeling methods rely on single-modal data, which limits their effectiveness and applicability. In this study, we introduce Uni-Clip, a contrastive learning framework that incorporates multimodalities, specifically structure and residue features of proteins, along with conformational and graph features of ligands. Through optimization with specifically designed CF-InfoNCE loss, Uni-Clip achieves comprehensive representations for PLIs. Uni-Clip shows superior performance in benchmark evaluations on well-established datasets, LIT-PCBA and DUD-E, achieving 147% and 218% improvement in enrichment factors at 1% compared to baselines. Furthermore, Uni-Clip serves as a practical tool for various applications in drug development, as demonstrated by virtual screening for a flat and challenging protein target GPX4, which identified potent inhibitors with an IC50 of 4.17 μM, and by target fishing for benzbromarone, underscoring the potential for repurposing benzbromarone in cancer therapy.