Optical properties of color centers in diamond have been the subject of intense research due to their promising applications in quantum photonics. In this work we study the optical properties of Xe related color centers implanted into nitrogen rich (type IIA) and an ultrapure, electronic grade diamond. The Xe defect has two zero phonon lines at 794 nm and 811 nm, which can be effectively excited using both green and red excitation, however, its emission in the nitrogen rich diamond is brighter. Near resonant excitation is performed at cryogenic temperatures and luminescence is probed under strong magnetic field. Our results are important towards the understanding of the Xe related defect and other near infrared color centers in diamond.