The best place to start learning about brain waves is probably the wikipedia article on Alpha waves:
https://en.wikipedia.org/wiki/Alpha_wave
From that page there are many many links to other articles.
So far as identifying relaxation goes, the best place to start would be the research paper "My Virtual Dream: Collective Neurofeedback in an Immersive Art Environment" by Natasha Kovacevic, Petra Ritter, William Tays, Sylvain Moreno, Anthony Randal McIntosh:
http://journals.plos.org/plosone/articl ... ne.0130129
According to this paper, relaxation can be determined as an increase in relative spectral power of Alpha. Doing a quick Google search for "EEG relaxation" will bring up many similar papers mentioning Alpha. You should note however that none of these theories are proven. Currently neuroscience is very much in it's infancy; there is a lot we do not understand and there is no simple guide which says frequency X means Feeling Y.