Few people visit Martha’s Vineyard just once. It’s a place that captivates visitors the second they step foot on the island. Just an hour by ferry from Cape Cod, Massachusetts, Martha’s Vineyard feels delightfully isolated, and while it is tiny (roughly 100 square miles), you could spend years discovering its dirt roads, craggy coastline and calm inlets. Most of the island has no dress code: Jeans and sneakers will do in just about every situation. But as you drive around, you’ll also find sprawling waterfront estates with sweeping ocean views, fine restaurants and upscale boutiques. It’s this laid-back-meets-luxurious atmosphere that has captivated everyone from the Kennedys and the Obamas to James Taylor and Bill Murray.

But ask anyone who spends significant time on Martha’s Vineyard, and they’ll tell you what most guidebooks leave out: That the best time to visit this idyllic summer town isn’t in summer at all. Rather, it’s the shoulder season, in May and September, when you’ll experience the very best the island has to offer.

Martha’s Vineyard is divided into two areas. Down-island, you’ll find Oak Bluffs, with its colorful gingerbread cottages, and Edgartown, with its gray shingle-style houses and white picket fences — a scene from a Vineyard Vines catalog if there ever was one. Up-island, the landscape is bucolic and rugged. More reminiscent of Ireland than New England, there are serpentine stone walls, horses roaming the hillsides and a rugged Atlantic coastline. To spend a perfect day on Martha’s Vineyard, you’ll want to experience both areas.