Indeed, it was the basis of my own speculation as to which one it could be. (Altho I remember this conversation being between Simon and Aka, not Xestris.)
By the way, Xestris points out the Chosen and the Crystal of Ivala as things that were created by our Tower manipulator. She didn't mention the third big miracle, the Wall of Ivala made during the TAW. Hester seemed to know about the process by which it was made, altho she later complained to Sarai about Chosen still appearing, so it's possible that they were indeed created by different people.
Anyway, if the Ivalan tower walker has already been introduced, let's review the likely suspects. We know it's not the Goddess of Magic since she was supportive of Ivala against her captor. It's probably not the Mother either, given how she acts, both in general and towards Ivalans in particular. The Sage of Ghenalon might actually be plausible. Xerces might have taken him out to take control of the Chosen, and lied about a second hidden towerwalker on Arclent as a red herring. Both unstable Chosen and the Crystal of Ivala appeared after this, right? So his war in the Tower would be about something else. It's a long shot, but heh, worth mentioning.
It's probably not Alonon, since the Anak mentioned two hidden towerwalkers on Arclent and he's likely one of them. Plus, he's terribly apathetic towards the world, while the Ivalan towerwalker mainly acts upon the world.
Now, let's examine the High Priestesses. We can assume it's not Sarai, for obvious reasons. Probably not Nabith either, she got her place quite late in the game. Since the Crystal of Ivala also talks to succubi and Chosen are lustful, we can also be reasonably sure that it's not little old Annah from Stineford. Plus, there was a red herring in the Tower in Stineford. There was also one in Aram, so I'd say that it's unlikely that either of its high priestesses is our culprit. (But if one is, I'd say it's Kerannii, based on personalities and such.) So the two most likely ones would be Hester and Bertricia. As said before, Hester was quite clearly behind the Wall of Ivala, but later seemed concerned about Chosen, so it's pretty muddy.
Let's also mention that the Crystal of Ivala's teachings to Ivalan succubi depends on the Council's result. If churches diverge, it asks them to join the Ardford church. If a new religion is created, it asks them to abandon it. So it seems to be quite accepting of succubi, but clearly favors the Ardford church. (Which is unsurprising given that it's located in Ardford, but it should still be pointed out.) Also, they built a new cathedral in Ardford specifically for this crystal, so that makes Alonon and Hester more likely suspects, since they're the ones with power to do that in this area. And Ardford is also the area with demon troubles since the very beginning! Conclusion: it's clearly Antarion. :p