Maintain stack. Randomly choose an operation to add a forward bracket or reverse bracket. When you add a forward bracket, add it to the stack. Add a reverse bracket only if there is at least one forward bracket in the stack. And in that case remove one forward bracket from the stack. In order to maintain a size limit, note that a bracket sequnce of size 2n has exactly n forward brackets. So use it to restrict the length of the sequence.

You can represent a regular bracket sequence as an Eulerian traversal of a tree — an opening bracket ( is a descent into a new vertex, a closing bracket ) is an ascent to an ancestor vertex.

Random trees can be generated using a Prüfer code. Then it becomes possible to generate a regular bracket sequence with equal probability.