The total population of terrorists ebbs and flows all the time. When the number goes up by one hundred, no one much notices. If the number goes up by one hundred because we release some previously identified terrorists, there is or will be a public outcry. But it's the same consequence.
Fewer terrorists are better than more terrorists, to be sure. But a terrorist we release is not obviously worse than a terrorist who was free in the first place.
We evaluate outcomes differently when we feel we are in control or should be in control. We should examine this intuition carefully, since it is not always justified.
We also treat an outcome differently when we feel it allows an enemy of ours to "get back at us." I suspect this difference in feeling is not usually justified and that it is the primary driver behind the fear of releasing terrorists.
I can think of "political theater" reasons why an attack from a released terrorist would be worse than an attack from an "already free" terrorist. Overall I do not yet feel that we are thinking about this issue rationally.