What is the opening dua in prayer?

Usually a short praise of Allah, said silently right after the first takbir. It's an optional sunnah, not a requirement. The most common version is "Subhanaka Allahumma wa bihamdika...", and the schools prefer slightly different wordings, so any authentic one is fine.

What it is and where it goes

After you say the opening "Allahu akbar" and fold your hands, and before you recite al-Fatiha, there's a short silent pause. The Companion Abu Hurairah once asked the Prophet ﷺ what he said in that pause, and he replied with a dua asking Allah to keep his sins far away, as far as the east is from the west (Sahih al-Bukhari 744). This is called the dua al-istiftah, the "opening supplication".

It is said quietly, even in prayers where the imam recites aloud, and it is only said in the first rakah.

The most common version

Many Muslims, especially in the Hanafi and Hanbali schools, say:

"Subhanaka Allahumma wa bihamdika, wa tabarakasmuka, wa ta'ala jadduka, wa la ilaha ghayruk."
Meaning: "Glory be to You, O Allah, and praise be to You. Blessed is Your name, exalted is Your majesty, and there is no god but You."

It is reported that the Prophet ﷺ said these words after the takbir when he stood to pray (Jami at-Tirmidhi 242). The Companion Umar ibn al-Khattab also used to say it aloud, which scholars often understand as a way of teaching it (Sahih Muslim 399). It's short, easy to memorise and only takes a few seconds.

Other versions

"Allahumma ba'id bayni wa bayna khatayaya...": "O Allah, set me apart from my sins as You set the east and west apart" (Sahih al-Bukhari 744).
"Allahu akbaru kabira, walhamdu lillahi kathira, wa subhanallahi bukratan wa asila": "Allah is greatest indeed, and much praise is for Allah, and glory to Allah morning and evening." A man in the congregation said this, and the Prophet ﷺ said he was amazed because the doors of heaven were opened for it (Sahih Muslim 601).
"Wajjahtu wajhiya...": a longer dua that begins with turning your face to the Creator of the heavens and earth (Sahih Muslim 771).

Part of that longer dua echoes the Quran:

قُلْ إِنَّ صَلَاتِى وَنُسُكِى وَمَحْيَاىَ وَمَمَاتِى لِلَّهِ رَبِّ ٱلْعَـٰلَمِينَ "Say, 'Surely my prayer, my worship, my life, and my death are all for Allah, Lord of all worlds.'" Quran 6:162

What the schools say

The schools differ on which wording to use. The Hanafi and Hanbali schools usually say "Subhanaka Allahumma...", the Shafi'i school often prefers "Wajjahtu wajhiya...", and the Maliki school generally does not recite an opening dua in the obligatory prayers. All of these are valid choices, so follow your teacher or mosque.

Do I have to say it?

No. It is optional, and your prayer is complete without it. If you're a beginner still learning al-Fatiha, skip it and come back to it later. And if you join the congregation late and the imam is already reciting, you can leave it out. To see how the prayer begins, see Why do Muslims raise their hands before praying? For the other words of the prayer, see What do Muslims say when they pray?

Key points

The opening dua is said silently after the first takbir and before al-Fatiha.
It is optional (sunnah), and only in the first rakah.
"Subhanaka Allahumma wa bihamdika..." is the most common version.
The schools prefer different wordings; all are valid.

Tip: Learn "Subhanaka Allahumma" first because it's short. After a few weeks, you can add a second version to use on other days.