An explain-plan is usually the best place to start whenever you have a slow query. To get one, run
DESCRIBE SELECT source_id FROM directions WHERE (destination_id = 10);
This will show you a table listing the steps required to execute your query. If you see a large value in the ‘rows’ column and NULL in the ‘key’ column, that indicates that your query having to scan a large number of rows to determine which ones to return.
In that case, adding an index on destination_id should dramatically speed your query, at some cost to insert and delete speed (since the index will also need to be updated).