We put forward an approach to the semantics of probabilistic program- ming centered on an action-based language equipped with a small-step operational semantics. This approach provides benefits in terms of both clarity and effective implementation. Discrete and continuous distributions can be freely mixed, un- bounded loops are allowed. In measure-theoretic terms, a product of Markov kernels is used to formalize the small-step operational semantics. A distinctive feature of this approach is that it directly leads to an exact sampling algorithm that can be efficiently SIMD-parallelized. An observational semantics is also in- troduced based on a probability space of infinite sequences, along with a finite approximation theorem. Preliminary experiments with a proof-of-concept imple- mentation based on TensorFlow show that our approach compares favourably to state-of-the-art tools for probabilistic programming and inference.