elementary number theory - Accidental divisibility sequence?

It seems (verified for $n \leq 3 \cdot 10^5$) that when the equation

$$p = n!+1 \bmod{\frac{n^2(n+1)^2}{4}}$$

is satisfied with some $n>1\in\mathbb N$ and $p\in\mathbb P$, it implies $n\in\mathbb P$ as well. The overall feel reminds me of a divisibility sequence like Fibonacci or $2^n-1$, which is the only time I've seen prime values locked to prime indices like that. However, I don't see many other factors being passed along with index factors, which is what I would normally expect.

Note that this applies only to the specific $p$ calculated as above using mod as an operation, and not to any arbitrary $p$ satisfying a congruence.

Can anyone explain why my sequence is behaving this way?

---

Upon reflection I think it has to do with my using Wilson's Theorem in a mangled way, and consequently I'm only getting non-$1$ values near primes and $2p$ semiprimes.

Some Mathematica code to show a table if you like:

upperBound = 100;
pretty[n_] := 
  If[Length@# > 1, CenterDot @@ #, First@#] &@
    FactorInteger[n] /. {{b_, 1} :> b, {b_, e_} :> Superscript[b, e]};
boldPrime[n_] := If[IntegerQ[n] && PrimeQ[n], Style[n, Bold, Blue], n];
Grid[Table[{n, boldPrime@pretty@n, 
   boldPrime@pretty@Mod[n! + 1, Plus @@ (n^2 (n + 1)^2/4)]}, {n, 2, 

   upperBound}], Alignment -> Right]

---

My updated thoughts:

For all $n>9$, it seems like $p=1$ except in one of three conditions:

1. $n+1$ is prime, in which case $n+1$ divides $p$


2. $n+1$ is an even semiprime, i.e. $2q$ with prime $q$, and in which case $2$ divides $p$


3. $n$ is prime, in which case $p$ may or may not be prime, presumably randomly.

If these are indeed the conditions (and I've verified them through $10^5$, so it seems very likely), that would be sufficient to explain the primality correlation. However, I'm still looking for a cogent explanation for why these particular conditions arise from my equation, and will happily accept an answer that can give me some insight into that.