Binary is just another positional system — base 2 instead of base 10. Only two symbols: 0 and 1.
That turns out to be the whole point. Building a reliable physical device with ten stable, distinguishable states is hard. Building one with two — voltage on, voltage off — is comparatively easy, cheap, and extremely reliable at enormous scale.
Every transistor in every chip is, at heart, a two-state switch pretending to be one of Leibniz's digits.