The Many-Colored Land by Julian May (1981) (cover art by Michael Whelan), volume one of the Saga of Pliocene Exile, is a science fiction book with some mythical aspects more commonly associated with fantasy. It was a 1982 Hugo Award nominee for best novel. I read the book some years ago and remember enjoying the entire series immensely.
In the 21st century Human Polity of the Galactic Milieu a portal is created in France allowing a one-way trip back in time to the Pliocene Epoch five million years ago. For years people -– adventurers, romantics, fanatics, conservatives, social misfits, the broken-hearted -- chose to exile themselves to the past. Many hope to find a simpler world. What they find is not at all what they expected.
There are two races of warring aliens, the tall, elf-like Tanu and the dwarf-like, gnome-like and ogre-like Firvulag. Both have seemingly magical powers called metaphysical talents. The books follow the fate of a group of exiles from 2110 who are split up upon arrival in the Pliocene.
The Many-Colored Land doesn't standalone. This first book in the series mainly sets up the story and introduces the characters. The rest of books in the Saga of Pliocene Exile are The Golden Torc, The Nonborn King, and The Adversary. Recommended.