Bosniak is right. Bosnian refers to a citizen of Bosnia & Herzegovina; Bosniak refers to a person of Bosnian ethnicity.
The CIA World Factbook says:
Definition Field Listing
Bosniak 48%, Serb 37.1%, Croat 14.3%, other 0.6% (2000)
note: Bosniak has replaced Muslim as an ethnic term in part to avoid confusion with the religious term Muslim - an adherent of Islam
The Washington Post uses the term, but always defines it like this: Bosniaks, or Bosnian Muslims.
The Congress of North American Bosniaks uses that term as well.
(title h/t Andy Carvin)