When you get to a battle with another ship you will be given some options to explore your ship or to engage in an overview. The other area aren't finished but eventually would be used for preparation. You don't really have to worry about them. Click engage in overview to go to the main combat mode.
You will appear above your ship with a white arrow floating over the water, this is your "cursor". You will also see yellow rings either side of your ship. These are your cannon aimers, they represent the maximum range of your cannons on either side, and the direction in which they are aimed. For maximum damage you will want to line the enemy ship up so that they are right on the ring or they are in between your ship and the ring.
If the overview it is your aim to either destroy the other ship or to board it and take it for your own.
There are several modes in the overview. Pressing the 'change weapon' key will cycle between these.
Move mode
Move mode is the mode you use to move your ship, it is shown by a white arrow in the bottom right corner of the screen. The white arrow represents your 'Cursor'. To move the cursor simply aim with the mouse and press w,a,s,d to move around. You press the attack button (left mouse click) to set a new waypoint. A waypoint is represented by a purple ring and it is where your ship will travel to. Setting a new waypoint when another is already active will override the old one. The angle your ship can turn at and the speed which it moves is relative to the type of ship.
Fire Mode
Fire mode is the mode you use to fire the cannons. It is shown by a set of cannonballs in the right bottom corner of the screen. Simply pressing attack (left mouse) will make the cannons fire. They will automatically choose which yellow icon to fire towards depending on which in nearer a ship. Stars will then appear on the left side of your screen, starting at 5 and going down. Once the stars have stopped this means your cannons are re-loaded and you can fire again. Firing when not all your cannons are reloaded will shoot a weaker volley of cannonballs. The more crew you have the faster you will reload.
Sail Mode
This is shown by an icon of sails in the corner. Clicking the left mouse button in this mode will cycling between raising and lowering your sails. With your sails down your ship will move slower but it will become more manourverable. Make sure you make use of this function tactically. It can be easy to become a sitting duck to cannon fire.
Shot type mode
Shown by a cannon of some shot types in the corner. In this mode you can you and of the new shot types that you may have purchased at harbor technicians. Simply left click and it will cycle through shot types to use. Chain shot will do most destruction to sails, grape shot will do high damage to crew on the enemy ship and round shot does the most damage to the hull of the enemy ship.
Boarding
Boarding an enemy ship is simply done by sailing into them, once this is done it will jump to a scene with yours and the enemy ship facing each other. There will be a plank and ropes allowing you to board the enemy ship and vice versa. To use the ropes simply walk into them and they will swing you over to the other ship. Reinforcements for both sides will keep coming when the number of men you/they have on deck gets too low. The battle will finish once one team has no men on the deck of their or your ship. If you die you will be given the option to go back on deck, but eventually you will give in and lose the battle.