We consider Bernoulli distributions and their regularizations, which are measures on the $p$-adic integers $\mathbb{Z}_p$. It is well known that their Mellin transform can be used to define $p$-adic $L$-functions. We show that for $p>2$ one of the regularized Bernoulli distributions is particularly simple and equal to a measure on $\mathbb{Z}_p$ that takes the values $\pm \frac{1}{2}$ on clopen balls. We apply this to $p$-adic $L$-functions for Dirichlet characters of $p$-power conductor and obtain Dirichlet series expansions similar to the complex case. Such expansions were studied by D. Delbourgo, and this contribution provides an approach via $p$-adic measures.
The preprint is available here.