I’d imagine it involving many many questions and answers, with links between them, modelling the division of the “big question” into its constituent chunks.
There are some big advantages — in particular, it’s easier to pay attention to individual parts, because there’s more structure than in blog comments.
As a first approximation, you might start out like this: Terry asks the polymath7 problem, linking elsewhere for motivation and background. Tim posts a first ‘answer’: “Could we attack this by proving Lemmas X, Y and then generalising the approach of Theorem Z?” and at the same time creates questions corresponding the Lemmas X and Y and a more open question about Theorem Z. Other participants can then go to those questions to give their thoughts. Answers don’t have to be “answers” in the convention sense — they’re just meant to correspond to “ideas”, and should often link to a new question if it’s obvious that the idea needs further development. The StackExchange software allows for comments on answers, which would allow short responses to previous answers.
The big disadvantages of StackExchange are that
* at this point, there’s no LaTeX support, although this will hopefully change.
* the reputation system inhibits new participants, at least at first (they can still ask and answer questions, but commenting and upvoting are limited).
* it may end up harder to understand the “big picture” than in a blog thread.
The solution to the first two of these may be to try a polymath project at mathoverflow.net itself, rather than a new installation. Many participants will already have reputation (and on an established site it’s very easy to gain enough reputation to comment and upvote, because any decent question will quickly garner reputation). It’s easy to filter questions by tags, so I think you could ignore everything else happening on mathoverflow.net if you wanted to.
The last problem might be addressed by having a “community wiki” answer at the top of each question, summarising progress so far, as well as regular progress reports on blogs.