During drilling for oil extraction purposes, the drill string experiences complex dynamic behavior, and this work delves into the experimental study and the mathematical modeling of such behavior. Self-excited vibrations, such as axial, lateral, and torsional vibrations, which can lead to detrimental effects such as bit bouncing, whirling, and torsional stick-slip are highlighted in this thesis. Distinct aspects of drilling dynamics are considered in this investigation to enhance the understanding of various phenomena. Initially, an experimental analysis of a lab-scale rig is conducted, providing valuable insights into the dynamics of such systems. And the influence of control parameters on the system’s response is examined, particularly in identifying the conditions under which the stick-slip phenomenon is likely to occur. Secondly, the thesis proposes system identification strategies for nonlinear systems, specifically focusing on the same laboratory test rig. An innovative ensemble approach is proposed, which combines gray and black-box modeling techniques to effectively calibrate the parameters of a dynamical system, particularly those associated with friction. This approach improves prediction accuracy compared to traditional gray-box methods while maintaining interpretability in the dynamic responses. Furthermore, the research employs physics-informed deep learning to estimate the low-dimensional model mechanical and friction parameters. Calibration using experimental data obtained from a specialized setup provides insights into the drill-string system s behavior. Finally, the thesis involves experimental investigations on the coupling between torsional and axial oscillations using a modified and adapted lab-scale drilling rig equipped with real drill bits and rock samples. In summary, this thesis advances our understanding of drill-string dynamics and presents helpful applications for system identification techniques.