After numerous attempts and hours spent on this issue I finally found the source of this problem.
Whenever you mangle with the appearance proxy ... things get bad. Simple change like:
[[UILabel appearance] setTextColor:[UIColor whiteColor]];
just causes the GUI to be unresponsive when zooming or panning. That beeing said it this post is the answer for my own question but I belive people can benefit from this information.