It is well known that on one occasion the eccentric Lord Orford drove his favourite team, consisting of four stags, from Houghton Hall, his country seat in Norfolk (after which, by the way, the Houghton meeting is called), into Newmarket, a distance of about twenty-nine miles. When he was approaching his destination, the Essex Hounds chanced to cross the road along which he had passed just before, and catching up the burning scent of the four stags, they immediately gave chase. As they drew near to the vehicle, their loud notes, or what fox-hunters call "their music", alarmed the stags, which galloped at full speed into the town, and dashed into the wide-open portals of the inn which stood on the site subsequently occupied by The Ram. The doors were immediately closed, and the lives of the stags saved from their eager pursuers. This occurrence was the cause of the name of the inn being changed to "Ram Inn", "ramish" being a word for strong scented.

The closed up archway on right side of frontage, now an entrance to the restaurant, still has plate signifying no right of way, from when it was another entrance to the yard.

The hotel was rebuilt by 5th Duke of Rutland - John Henry Manners in 1815 and became "The Rutland Arms Hotel". The builder was John Kent who also designed the New Jockey Club
The Bury and Norwich Post of 13th September 1815 has "Ram Inn", and on October 13th it is referred to as "Rutland Arms Inn.

The hotel was purchased in 2018 by the owners of the Bedford Lodge Hotel who planned to restore it to its previous glory

However, having closed the hotel for renovations, including demolishing the wing the other side of Palace Street and the access bridge, legal difficulties prevented further work. All something to do with a covenant on the land the other side of Palace Street.
As a consequence the entire premises are on the market again (2023)and the High Street has yet another vast empty building. Eventually sold by Review Hotels Ltd in January 2024