We theoretically and experimentally investigate nonlinear Zeeman (NLZ) effects within a polarised single-photon source that uses a single 87Rb atom strongly coupled to a high finesse optical cavity. The breakdown of the atomic hyperfine structure in the ${{\rm{D}}}_{2}$ transition manifold for intermediate strength magnetic fields is shown to result in asymmetric and, ultimately, inhibited operation of the polarised atom–photon interface. The coherence of the system is considered using Hong–Ou–Mandel interference of the emitted photons. This informs the next steps to be taken and the modelling of future implementations, based on feasible cavity designs operated in regimes minimising NLZ effects, is presented and shown to provide improved performance.