HMS Montrose moved between the three boats and the tanker British Heritage before issuing verbal warnings to the Iranian vessels, a spokesman said.
He described the Iranians' actions as "contrary to international law".
Iran had threatened to retaliate for the seizure of one of its own tankers, but denied any attempted seizure.
Boats believed to belong to Iran's Islamic Revolution Guard Corps (IRGC) approached the British Heritage tanker and tried to bring it to a halt as it was moving out of the Gulf into the Strait of Hormuz.
Guns on HMS Montrose, the British frigate escorting the tanker, were reportedly trained on the Iranian boats as they were ordered to back off. They heeded the warning and no shots were fired.
The BBC has been told British Heritage was near the island of Abu Musa when it was approached and harassed by the Iranian boats.
Although the island is in disputed territorial waters, HMS Montrose remained in international waters throughout.
A UK government spokesman said: "Contrary to international law, three Iranian vessels attempted to impede the passage of a commercial vessel, British Heritage, through the Strait of Hormuz.
"We are concerned by this action and continue to urge the Iranian authorities to de-escalate the situation in the region."
What does Iran say?
Quoting the public relations office of the IRGC's Navy, the Fars news agency said, in a tweet, the IRGC "denies claims by American sources" that it tried to seize British Heritage.
"There has been no confrontation in the last 24 hours with any foreign vessels, including British ones," the IRGC added, according to the AFP news agency.
Iranian foreign minister Mohammad Javad Zarif said the UK made the claims "for creating tension".
"These claims have no value," Mr Zarif added, according to Fars.
Why are UK-Iran tensions escalating?
The relationship between the UK and Iran has become increasingly strained, after Britain said the Iranian regime was "almost certainly" responsible for the attacks on two oil tankers in June.
Last week, British Royal Marines helped the authorities in Gibraltar seize an Iranian oil tanker because of evidence it was heading to Syria in breach of EU sanctions.
In response, an Iranian official said a British oil tanker should be seized if its detained ship was not released.
Iran also summoned the British ambassador in Tehran to complain about what it said was a "form of piracy".
On Wednesday, Iranian President Hassan Rouhani mocked the UK, calling it "scared" and "hopeless" for using Royal Navy warships to shadow a British tanker in the Gulf.
HMS Montrose had shadowed British tanker the Pacific Voyager for some of the way through the Strait of Hormuz, but that journey had passed without incident.
"You, Britain, are the initiator of insecurity and you will realise the consequences later," Mr Rouhani said.
The UK has also been pressing Iran to release British-Iranian mother Nazanin Zaghari-Ratcliffe who was jailed for five years in 2016 after being convicted for spying, which she denies.
Iran appears to have been attempting to make good on its threat against British-flagged vessels in the wake of the seizure of an Iranian tanker off Gibraltar.
But though this incident has a specifically bilateral dimension, it is also a powerful reminder that the tensions in the Gulf have not gone away.
And with every sign that the dispute over the nuclear agreement with Iran is set to continue, things may only get worse.
The episode may add some impetus to US-brokered efforts to muster an international naval force in the Gulf to protect international shipping.
But most worrying of all, it shows that elements within the Iranian system - the Revolutionary Guard Corps's naval arm, or whatever - are intent on stoking the pressure.
This inevitably plays into President Trump's hands as Britain and its key European partners struggle to keep the nuclear agreement alive.
The US has blamed Iran for attacks on six oil tankers in May and June.
The chairman of the US military's Joint Chiefs of Staff said, on Wednesday, it wants to create an multi-national military coalition to safeguard waters around Iran and Yemen.
The Trump administration - which has pulled out of an international agreement on Tehran's nuclear programme - has reinforced punishing sanctions against Iran.
Its European allies, including the UK, have not followed suit.
Iran's ambassador to the UN has insisted Europeans must do more to compensate Tehran for economic losses inflicted by US sanctions - otherwise Iranians will continue to exceed limits on their nuclear fuel production.