Following "comprehensive screening" of the Cambo field, Shell "concluded the economic case for investment in this project is not strong enough at this time, as well as having the potential for delays," the company said in a statement.
Private equity-backed Siccar Point, which owns a majority stake in the field, confirmed in a separate statement that "Shell has taken the decision to not progress its investment at this stage."
The Cambo project off the Shetland Isles has been at the centre of a political debate on whether Britain should develop new fossil fuel resources as it seeks to become a net zero carbon economy by 2050.
Climate activists have pointed to a report by the International Energy Agency (IEA) saying that no new oil and gas projects should be developed in order to restrict global warming to 1.5 degrees Celsius.
"Cambo remains critical to the UK's energy security and economy," Siccar Point Chief Executive Officer Jonathan Roger said in a statement.
Britain's government has also responded to the IEA report by saying energy security is important. During the COP26 climate conference which was held under British auspices last month, Britain also declined to join an alliance of countries pledging to stop new oil and gas developments on their territory.
"Whilst we are disappointed at Shell's change of position ... we will continue to engage with the UK Government and wider stakeholders on the future development of Cambo," Roger said.
Adam Matthews, chief responsible investment officer at Church of England Pensions Board, a shareholder which leads engagement with Shell over its energy transition strategy, welcomed the move.
"The message is clear to the U.K. Government (that ultimately decide if the field is exploited) that companies beginning to transition will not allocate the capital to such projects," Matthews said in a tweet.
Friends of the Earth, an activist group which won a climate court case against Shell in the Netherlands this year applauded the move.
"The future of the project is now in serious doubt - as it should be. There is no need for a new oil field during a climate crisis," the group said on Twitter.
Shell owns 30% in the project, while Siccar Point, which operates it, holds the remaining 70%. The field could produce up to 170 million barrels of oil equivalent and 53.5 billion cubic feet of gas over 25 years, according to Siccar Point.
It was unclear whether the field could be developed without Shell's support.
Shell faced another setback for its plans in the British North Sea in October when a regulator rejected plans to develop the Jackdaw gasfield after considering its environmental statement, industry sources said.
(Reporting by Ron Bousso and Shadia Nasralla; Editing by Mark Potter, Aurora Ellis and Marguerita Choy)
By Ron Bousso and Shadia Nasralla