As recorded in Village records, dated May 11, 1975 by Historian Samuel B. Cross, Frank C. Rogers (1857 - 1921), with his partner the father of Ward Melville of Stony Brook, bought the property called the Beach Club in 1880. He married, Carrie Larabee (1862 - 1944) on January 22, 1882. On May 1, 1882 they purchased the property on the south side of Main Street and the east side of Seafield Lane. Sometime between 1873 and 1888, Seafield House was built. Frank was the brother of George Rogers, the grandfather of Ed Rogers, Jr.

A local historian, Charles McDermott was a one-time owner of Seafield House. He was the editor of the Long Island Formm newspaper. He donated his collection of historical books to the Westhampton Library.

Two residents of New York City, Kenneth Ryan attorney and John Ringsted, owned Seafield House at different times. Each used it as a summer home.

1972/82 - The Collins Family lived here as year round residents. In March of 1982, Mrs. Elsie Collins was encouraged by friends to start a Bed and Breakfast. The Seafield House was ideal because of its conduciveness to an atmosphere of warmth and charm. There is a wonderful feeling of well-being to all who enter her doors. Mrs. Collins has preserved and enhanced this feeling by filling it with her lovely antiques and personal touches.