Around 50,000 ethnic Serbs who live in Kosovo refuse to recognise Pristina's authority, and still consider themselves a part of Serbia.
Belgrade has said it will never recognise Kosovo's independence.
Hundreds of police officers, judges, prosecutors and other state workers from the Serb minority quit their jobs this month after Pristina ruled that local Serbs must finally replace car plates issued by Kosovo Serb municipal authorities, loyal to Belgrade, with Kosovo state ones.
EU foreign policy chief Josep Borrell, who chaired the talks, blamed mainly Kosovo for rejecting an EU proposal on how to resolve the dispute.
NATO, which has around 3,700 peacekeepers in Kosovo, said it was ready to intervene should the security situation be threatened.