These issues were related! Also it fixed a problem I didn't even know I had, and I pretty much created a new best practice that not even the Freenas folks know about (as I was on their forums and they had no advice other than to blame my hardware)
Come to find out I had performance issues all along with FreeNAS and never knew it because the 1gb interface was not fast enough to really expose the issue.
Upgrading to 10gb caused actual packet loss due to the problem, and made me start having to investigate. Ends up the Plex issue was a FreeNAS issue with how it bridges network interfaces, and by fixing Plex it also fixed my slow speeds on my 10gb link, AND it also fixed the sluggish response to my jails.
Now everything is so snappy and fast!
As a bonus I was able to turn security back on for my vSwitch as I no longer need promiscuous mode or forged transmits.