The changes are the latest churn on the board as Son focuses on technology investing through his Vision Fund unit over taking majority stakes in companies.
Fisher was an architect of SoftBank's disastrous bet on office space firm WeWork, while Segars' firm is to be sold to U.S. chip firm Nvidia Corp pending regulatory approval.
Fisher's decision to step down was his own, a SoftBank spokeswoman said without providing further details.
SoftBank has nominated the chair of gaming firm Koei Tecmo Holdings Co Ltd, Keiko Erikawa, and the co-CEO of group internet business Z Holdings Corp, Kentaro Kawabe, to join the board.
The group has also nominated Ken Siegel, managing partner at the Tokyo office of law firm Morrison & Foerster, who worked on deals including the Arm sale and the sale of U.S. robotics firm Boston Dynamics to South Korea's Hyundai Motor Co.
The nominations will be put to a shareholder vote at the conglomerate's annual general meeting next month.
Waseda University business professor Yuko Kawamoto is to depart the board after just a year to join a government agency.
"I believe it is rare for a CEO to listen to others so attentively," Kawamoto said in a statement, adding that "it would be nice if the 'obligation to dissent' or to disagree when necessary, was more widespread throughout SBG."
The group's "biggest challenge for the future is to design a succession plan," Kawamoto said, pointing to the recent announcement of the successor to Jeff Bezos, the 57-year-old CEO of Amazon.com Inc.
Son, 63, is yet to set out such a plan at SoftBank, which earlier this month booked record annual profit at its Vision Fund unit on valuation gains.
The board changes maintain its membership at nine people with Koei's Erikawa to be its only female member.
SoftBank cut the number of board members from 12 to amplify the voices of external directors following investor pressure for greater oversight at the group.
(Reporting by Sam Nussey; Editing by Jacqueline Wong and Christopher Cushing)
By Sam Nussey