Finlay Macleod of the Fishermen’s Mission said all four reported missing had made it to their life raft.
But two of them tried to swim for the shore, which was in sight.
One of the four was found safe, while another is still missing.
Two of the four men on the raft were later found to have died, whilst another is still missing.
Mr Macleod told the BBC: “Two of them decided to swim to the shore, which they could see.
“And during that swimming, one of them drifted away from the other and he was lost.”
He added: “The other one made it to the shore, and after clinging to the rocks for some considerable time he was rescued by the boat from the RNLI.”