Seven & I, which runs the 7-Eleven convenience store chain, had been in exclusive talks to buy the Speedway business for $22 billion, the Nikkei newspaper and Bloomberg News had reported.
The company considered the deal but decided against it because of worries about the price, said the source, who is familiar with the matter, but sought anonymity as he was not authorised to speak to reporters.
The Nikkei, which first reported the deal was off, also cited growing concerns about a global economic slowdown due to a coronavirus.
The negotiations came after Marathon, under pressure from activist investor Elliott Management, said last year that it would launch a sweeping restructuring, including spinning off its Speedway retail business, which it said was worth as much as $18 billion, including debt.
Elliott had pushed for Marathon to split into three companies, for a boost of as much as $40 billion to shareholder value.
For Seven & i, the deal would have added to a portfolio of more than 1,000 gas stations and corner stores in the United States that it acquired in 2017 through a $3.3 billion deal with Sunoco.
But both analysts and investors had warned the reported $22 billion price was high. Shares in Seven & i fell more than 15 percent since the talks were first reported, outstripping a decline of 9 percent in the Nikkei 225 average.
Standard & Poor's had also said a deal at $22 billion would put pressure on Seven and i's credit rating.
Japanese companies are seen vulnerable to overpaying for assets as they are desperate to expand overseas, given years of slow economic growth and shrinking demographics at home.
But analysts have said it still made sense for Seven to seek expansion in north America.
Japan is considered a saturated market for convenience stores, with competition from rivals such as FamilyMart Co Ltd and Lawson, as well as discount drugstores and online retail giants like Amazon.com Inc.
The stores also face pressure in Japan to review the 24-hour policy, with many franchise owners struggling to keep stores staffed amid a labour crunch.
7-Eleven stores originated in the United States but Japanese retail executive Toshifumi Suzuki went from an initial tie-up to a ubiquitous chain selling everything from drinks and ready-made lunches to neckties and underwear.
He began expanding the stores throughout Japan in the 1970s, and their 24-hour policy and franchise system proved a good match for a round-the-clock work culture and dense population.
Seven & i Holdings, which also owns Ito-Yokado supermarkets, later rescued the U.S. business, which went through bankruptcy.
(Reporting by Ritsuko Ando and Ritsuko Shimizu; Writing by David Dolan; Editing by Susan Fenton and Clarence Fernandez)