number theory - Prime dividing binomial coefficient involving prime power

I was wondering if there was a straightforward proof of the following fact (which I can show is true for specific cases, but not generally):

Let $n$ be composite, and let $p$ be a prime factor $n$. Suppose $p^k$ is the highest power of $p$ that divides $n$. Then the following is true:
$\binom{n}{p^k} \ne 0 \pmod p$.

My approach was to see that $\binom{n}{p^k} = \frac{n(n-1)\ldots (n - p^k + 1)}{(p^k)!}$. However, how do I show that $p$ does not divide this quantity? It seems not obvious how to argue that $p$ occurs the same number of times in the numerator as in the denominator.

**Update: ** I am looking for an elementary proof "from scratch" that does not rely on Lucas' Lemma or anything too heavily reliant on field theory. Preferably an argument that pops out from analyzing the binomial coefficient. Thanks!