Your very likely losing a bit of output by the flexing. Longterm it can also damage your box, so I'd try and nip it in the bud now, how you do that is up to you..
Bracing works best when it's coupled to a wall that is perpendicular to it.. Very hard to stop a baffle from flexing back and forth when the front baffle is what your connected to. Get some 2x4's and make a T style brace running from the center of each panel. Your sides are smaller and are moving much less and in a totally seperate direction as your back, by coupling to them you can stop the whole thing from moving, since those sides do NOT want to move back and forth, they want to move side to side lol. Imagine trying to pull apart a piece of birch by tugging it apart longways, not going to happen...
I know a lot of guys dont' like bracing since it hurts airflow and in a perfect world I'd agree. However, building 6 layer boxes gets stupid heavy and isnt' practical for daily. (not that wall are all THAT practical to begin with, but still lol)
Fiberglass works, but it's a lot of work to get what you can get out of 15 minutes worth of work with some good bracing.