City lost 2-1 at home to title rivals Tottenham and lie in fourth, six points off leaders Leicester.
A contentious Harry Kane penalty gave Spurs the lead, before Kelechi Iheanacho equalised, but Christian Eriksen scored the winner late on.
“There is not a chance we will give up on the title. We have to improve,” City captain Kompany told BBC Sport.
Manuel Pellegrini’s side have now lost twice in three Premier League games – against Leicester and Spurs – since announcing the Chilean will be leaving the club in the summer to be replaced by Pep Guardiola.
City have won the title twice in the last four seasons, coming from eight points behind to pip rivals Manchester United on goal difference in 2010-11 and beating Liverpool by two points in 2013-14, having been seven points adrift.
“We probably could have played a little bit better. Until the penalty, the game plan was OK and we were defending well,” added centre-back Kompany.
“But we would be stupid to forget our history and how we react in those moments.”
City boss Pellegrini was unhappy with referee Mark Clattenburg’s decision to award a penalty for the first goal, as the ball hit Raheem Sterling on the back as the winger jumped to block Danny Rose’s cross.
“It was the key moment that decided the game,” Pellegrini said. “It was the same referee in the first game where they were two clear goals in offside positions and we lost 4-1.
“We are going to continue fighting until the end. We still have 36 points to play for and are only six points behind the leaders. We will not give up and continue working and improving.”
It was a good day for both north London sides as Tottenham followed up Arsenal’s last-second victory over Leicester at lunchtime. Both team are now on 51 points, just two behind the Foxes.
“I think that it was a game that was very important. You could feel it was not a normal game and a real test for us. For that we feel very pleased for the players,” said Spurs boss Mauricio Pochettino.
Asked if his side can win their first league title since 1961, the Argentine replied: “I don’t like to speak abut the future, it is more important for us to keep working hard and to show we can win in this kind of stadium.
“The supporters are right to dream. They show big support from the beginning of the season and I say thank you to them. It is important for our supporters to believe.”