Britain's biggest drugmaker said on Thursday that it would continue with the Phase III trial of its MAGE-A3 therapeutic vaccine to see if it benefited a subset of patients with a particular genetic signature.
The trial has two joint targets, known as co-primary endpoints, looking at both the broad population and a patient subset, so it is possible the treatment will yet prove effective for a small number of people with the deadly skin cancer.
It is also being tested in lung cancer.
The vaccine is one of two particularly high-risk developmental GSK products believed by industry analysts to have the potential to become multibillion-dollar sellers. The other is a heart drug called darapladib, designed to fight clogged arteries in a different way to statins.
In both cases, however, analysts have been wary of modelling firm forecasts of sales, given the above-average risk of failure. GSK stock was down 0.8 percent by 0835 GMT.
Citigroup analyst Andrew Baum said that already low consensus expectations for the product are now likely to fall further. He currently forecasts sales of only 149 million pounds in 2018 for the MAGE-A3 vaccine in both melanoma and lung cancer. That figure is about half the industry consensus.
Unlike traditional preventative vaccines, the MAGE-A3 treatment is designed for people with established disease, helping their immune systems to prevent the return of disease after surgery.
Other companies, including Bristol-Myers Squibb, Roche and Merck & Co, have had some recent notable successes in clinical trials of innovative drugs to boost the immune system, but GSK is still pushing the scientific boundaries with its MAGE-A3 vaccine.
Cancer vaccines have proved difficult to develop over the years. A different vaccine for lung cancer from Germany's Merck KGaA also failed in a clinical test last December.
GSK's vaccine was developed for use in patients with tumours that express the cancer-related MAGE-A3 gene, which occurs in about 65 percent of Stage III melanomas.
It failed to extend disease-free survival significantly when compared with placebo in this population in the Phase III study.
The second endpoint of the trial is to discover whether disease-free survival is improved in a subset of MAGE-A3-positive patients. Results from this analysis are expected in 2015.
GSK added that no safety issues had emerged in the trial.
"We remain committed to identifying a patient sub-population who may benefit from this investigational treatment," said Vincent Brichard, head of immunotherapeutics at GSK Vaccines.
A spokesman said it was not yet possible to say what proportion of patients might fall into this subset.
GSK is also continuing to study MAGE-A3 in another Phase III study against non-small cell lung cancer. The first data from this trial are expected in the first half of 2014.
Alistair Campbell, of Berenberg Bank, described the chances of a success in lung cancer as "slim".
"This asset was widely regarded as a 'wild card' by the market. While there had been some hope that this approach might work, the result is not a surprise," he said in a research note.
Although the news on melanoma was disappointing, Campbell noted that 2013 had still been a strong year for GSK research, with new drug approvals in HIV, cancer and respiratory disease.
GSK is now waiting to hear if its latest lung drug, Anoro, will be approved. A U.S. Food and Drug Administration advisory panel is set to assess the medicine on September 10.
(Editing by David Goodman)
By Ben Hirschler