Idlemoor: The History Of 'Sconset's "Haunted House"

Local legend has it that 11 Baxter Road, one of the oldest houses on the Sconset Bluff, is haunted. At least it looks like it could be. But even if there is no truth to the ghost stories, the old house on the eastern edge of the island has an interesting history.

The building - perhaps one of the most photographed on the island - is well-known for its architectural style, its slowly deteriorating facade, and the ghostly aura that comes with it.

The story of Idlemoor begins in 1873 with a man named William Flagg. In the late 19th century, Flagg acquired large swaths of ‘Sconset. His deeds stretched to the foot of the bluff, giving him control over the entirety of what would later become the famed ‘Sconset Bluff Walk. Indeed, Flagg was responsible for the footpath that even now curls along the edge of the bluff, overlooking the ocean.

But Flagg did not purchase the land simply to make a path. He saw a future where the then-undeveloped bluff was lined with houses. And as his holdings grew, he began to subdivide his property and sell it to buyers who might wish to build homes in ‘Sconset. One of those buyers was Abraham Rice, a financial manager for the Detroit Safe Company. In 1884, Rice bought the plot of land that would become 11 Baxter Road, and in 1885, Idlemoor was built.

Rice was enamored with the home and even considered making it his permanent residence. He wasn’t the only one impressed: The Inquirer & Mirror reported on it favorably, and it has gone down as a notable piece of ‘Sconset architecture, the subject of a lengthy passage in Margaret Moore Booker and Pat Butler’s book “Sea Captain's Houses & Rose Covered Cottages.”

Idlemoor is a notable example of the Stick architectural style, named for its use of overlay board strips to imitate an exposed frame. It has also been compared to the so-called “gingerbread houses” of Oak Bluffs, Martha’s Vineyard. This is perhaps because the architects, EA & MB Leighton, were from Oak Bluffs, then known as Cottage City.

The Rice family was one of the first to make ‘Sconset their summer residence, and by all accounts, they enjoyed their time on the island. The popular name of the house, after all, is derived from its beautiful view of the moor and the idle nature of the family’s summer stays at the residence.

After Abraham died, his daughter Abby Rice Fyfe - alternately spelled Abbie - inherited the property in 1911. Fyfe is most well-known for donating the land for the ‘Sconset Casino, which she also inherited from her father. In 1928, Fyfe sold the land to Edmund Maginess Barnum for $800, equivalent to around $14,500 today. While land wasn’t worth nearly as much in ‘Sconset then as it is now, it’s likely Barnum got a bit deal on the property: his father was married to one of Fyfe’s sisters.

Barnum was a soldier like his father, eventually achieving the rank of Colonel. He served and died in the European Theater of World War Two, passing away in 1943 at the age of 51. His remains are buried in London with other fallen soldiers. When he died, his wife Elizabeth Barnum took ownership of the house. The property stayed in her family until 1972 when it was purchased by Wilbur and Nancy Davis. The Davises became the first people to ever own Idlemoor who were not related by blood or marriage to the original owner, Abraham Rice. For nearly 100 years, the house stayed in his family, passing from generation to generation as Nantucket developed from a sleepy fishing village to a tourist destination.

If anyone is haunting Idlemoor, they are probably related to Rice, though it is also worth noting that in the entire history of Idlemoor, no one has ever lived there year-round.

Idlemoor last changed hands in 2019, when Bruce Davis inherited it from his parents. Davis did not return several messages from the Current seeking to discuss Idlemoor.