At Nassau, there are four ships labeled as merchants and two other ships. The merchants are not hostile until you approach, while the other ships are red right from the start, but will not move until you approach. You must successfully board one of the merchants to complete the mission; boarding one of the others does not count.
The various hostile ships range in level from 1 to 5. As you're fighting them 1 on 6, you have little hope of winning a protracted battle. Instead, you must pick one ship, attack it, wear it down enough to win in boarding combat, and board it all before the rest of the ships can overwhelm you.
It is easiest to take on the nearest ship, as this gives you a bit of time to fight before the other ships arrive. You can enter the mission, check to see if the nearest ship is level 1, and if not, exit and try again. Rerolling until you get a level 1 ship can make this mission much easier.