Police, ambulance and three fire engines were called to the scene on Town Bridge just after 4.30pm.
It is believed the man, described as in his 20s, was wearing a wetsuit and had jumped in the River Nene 'near to the library and town bridge'.
He didn't appear to be in distress and one passerby said he 'had his thumbs up'.
Fire crews confirmed around 5.30pm that the man had been 'brought to safety out of the river'.
He was seen floating down the river towards the bridge, prompting a passerby to alert emergency services.
One woman, who did not wish to be named, said: "He was just seen floating in the water towards Elywn Road and it was quite frightening as he was spotted under the bridge.
"Someone said they managed to get him out but then he jumped back in again.
"It's frightening to think that a young person has had to do something like this."
Traffic in Broad Street was at a standstill while people gathered on the bridge to watch.
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules here