Aspen Art Museum, Colorado, USA: The museum dividing America's wealthy resort town

When Shigeru Ban first arrived at the site of the proposed new Aspen Art Museum he was nonplussed. Not a fan of monumental architecture, initially he wasn't interested in the nondescript site. It wasn't until the Pritzker prize-winning architect was taken up high in a neighbouring building, that it made sense to him. From there he devised a solution for the downtown Aspen site that featured no ground floor aspect, and required the maximum gallery space. He did this by inverting the entrance and setting it on the rooftop. Which is perhaps not surprising, given the swanky ski town is all about the Rocky Mountains, which majestically surround Aspen itself.

In an ideal world, the Japanese architect, wants everyone to start from the top of the minimalist style building, basically a glass box nested in a resin infused paper lattice screen and topped with a timber truss roof, and work their way down, just like you would on a ski slope. Arriving visitors are meant to ascend, as on a ski lift or gondola, via a huge glass elevator, or a grand sweeping staircase, to the uppermost level. Heidi Zuckerman Jacobson, the AAM's CEO and curator, admits in practice, you can't compel people to follow the architect's reversed sequence. Many visitors, as we see firsthand, start at the bottom with the first gallery they enter at street level.

A huge fan of Ban, Heidi was intrigued by the architect's submission from the get go. Following a lengthy selection process, and a trip to Ban's Tokyo office, the Japanese architect was unanimously chosen for the project eight years ago - before he became known as a prodigy of humble architecture and humanitarian building structures in the wake of natural disasters. The $45 million museum, which replaces the 35-year-old AAM on the banks of the Roaring Fork River, is an astounding mesh of materials that pushes the envelope. It's Ban's first American museum, and one that has divided the wealthy resort town.

When I meet the petite and dynamic Zuckerman Jacobson, tanned and wearing an eye catching red and white dress, she shows no visible sign of the strain of opening the contentious museum. Everyone in the town, with its charming Western Victorian buildings and unremarkable ski condominiums, has an opinion of the boxy 33,000-square-foot museum spread over three levels. "I like it's contemporary look; it's cool," a hotel concierge tells me, but he appears to be in the minority. Others claim the building is out of scale to the small town's brick-faced, late 19th century buildings. Some detest the lattice like exterior and in the latest controversy the museum is in hot water over an exhibition featuring live tortoises wearing iPads. Whichever way you look at it, it's got everybody talking.

When I visit, on a blue, cloudless day, shortly after the museum's opening on August 9, we head straight for the top, just the way Ban intended. Tables of men and women are happily lunching at the rooftop So café with its expansive views of Aspen Mountain and Independence Pass. A rooftop sculpture garden offers unhindered views of Aspen's main ski slope, and in addition to the café there's a bar and an outdoor screening space. Most impressive is the exposed three-dimensional, truss ceiling – a remarkable feat of architecture, woven on site with not a single nail or screw.

From Aspen's only public rooftop space, Jacobson explains the complexities of assembling the woven screen exterior onsite, a paper tube façade that Ban designed to give a sense of movement and to mimic the imperfections in old brick. "It seems like a grid but it's actually not," she tells us as we descend the grand staircase, where a glass partition divides the lattice screen exterior and the interior. Cleverly the staircase feature snow-melt stairs and while you ascend or descend you can see the mountains around you. It provides both exterior access to the public roof and interior access to the galleries.

The six galleries themselves are purposefully bland: white walls and not a single switch or sensor. Natural light features in four of them. The museum will have no permanent collection, and when we visit an exhibition of Ban's humanitarian work is on show. This is the one that most piques my interest. True to size replicas of Ban's groundbreaking designs for refugee shelters, and the design for Christchurch's Cardboard Cathedral are on display. Visitors are able to walk around as well as enter four of Ban's innovative designs for humanitarian relief – most of which he funds, or find funds for, himself.