Dick Advocaat resigned on Saturday as Serbia coach following the team's 3-1 defeat against Denmark in a European Championship qualifying match.
Serbia's Football Federation said in a statement that Advocaat decided to leave the post because he felt there was nothing he could do to improve the team's performance and self-confidence.
Advocaat took the post in July with the aim to lead Serbia to the 2016 European Championship in France. But Serbia are currently joint bottom-placed in Group I with just one point after their first three matches.
Nicklas Bendtner scored twice as Denmark came from a goal down to beat Serbia 3-1 in Belgrade on Friday.
The match was played in an empty stadium because of incidents in a qualifier between Serbia and Albania last month.
Serbia were penalised by UEFA for pitch skirmishes involving players and fans over an Albanian flag that was flown above the stadium by a drone.
In a statement released by Serbia's football federation, Advocaat said, "I am very sorry our joint project ended like this." He added that the pressure was "too big for this team."
"Like never before in my career, I was faced with a problem I could not solve," Advocaat said.
Advocaat has also coached the national teams of United Arab Emirates, Belgium and Russia, as well as clubs such as PSV Eindhoven and Rangers. Last season, he had a second stint with Dutch club AZ Alkmaar, who reached the Europa League quarterfinals.