Discussions over the potential tie-up, which would have increased competition for Russia's largest lender Sberbank and smaller rivals in technology and banking, stalled less than four weeks after they were announced by Tinkoff parent TCS Group Holding.
TCS said in a statement it had agreed with Yandex not to proceed with the transaction, but Tinkoff would continue to partner with it on existing and future projects.
"Today I decided to break the possible deal with Yandex," TCS founder Oleg Tinkov said in an internal email which was seen by Reuters on Friday.
Tinkov, a major TCS shareholder, could not be reached for comment and the online bank's press office declined to comment.
Yandex shares fell 2.2% in Moscow to 4,652.0 roubles ($59.64) at 1451 GMT, while TCS Group's London-listed shares were 1.3% lower, after earlier hitting a three-month low.
'NOT FOR SALE'
TCS and Yandex announced they were in talks on Sept. 22, a few months after Yandex said it was ending its partnership with Sberbank, Russia's largest lender.
Tinkov said in his email that Yandex was interested only in a takeover of Tinkoff, the world's largest fully online bank with more than 10 million customers across Russia, whereas Tinkov viewed the deal as a merger.
Yandex had agreed that Tinkov would be involved in managing the bank and help Yandex in general after the deal, Tigran Khudaverdyan, Yandex's deputy CEO, said in a post on his corporate blog, confirmed as genuine by its press department.
Khudaverdyan said Tinkov had made more demands after each stage of talks and his decision did not come as a surprise.
Tinkov had previously played down talk of an acquisition, insisting the deal being discussed with Yandex was not a sale but a merger.
"Tinkoff is not for sale, neither to Yandex, nor MTS," Tinkov said, referring to reports that TCS was in discussions to sell the bank to Russia's top mobile operator MTS.
And Vladimir Yevtushenkov, head of Sistema, MTS' major shareholder, told Reuters he not been approached by Tinkov.
Maria Sukhanova, a TMT analyst at BCS Global Markets, said further approaches were possible.
"We can't rule out that parties will resume talks in the future or that another buyer in the TMT universe for all of, or a stake in, TCS will ultimately emerge," she said.
(Reporting by Andrey Ostroukh, Alexander Marrow, Anastasia Teterevleva and Tatiana Voronova; additional reporting by Polina Devitt, Nadezhda Tsydenova and Elena Fabrichnaya; Editing by David Goodman and Alexander Smith)
By Alexander Marrow and Andrey Ostroukh