Quantum memories, capable of storing single photons or other quantum states of light, to be retrieved on demand, offer a route to large-scale quantum information processing with light. A promising class of memories is based on far-off-resonant Raman absorption in ensembles of Λ -type atoms. However, at room temperature these systems exhibit unwanted four-wave mixing, which is prohibitive for applications at the single-photon level. Here, we show how this noise can be suppressed by placing the storage medium inside a moderate-finesse optical cavity, thereby removing the main roadblock hindering this approach to quantum memory.