Selecting advisors...

Your advisors should be career builders, not career talkers. In other words: ask yourself if they actually built and operated anything successful themselves?
Your array of advisors should be domain experts - one for each of the hard problems you're trying to solve. If you are in e-commerce, supply chain, and pool supplies, then you typically need an advisor(s) in each category. Sometimes you get lucky and find unicorns that have done a few of the categories you need - bonus!

Managing advisors...

Set up a series of standing meetings at a regular cadence.
Bring your biggest/hardest questions and opportunities for them to work on.
Ask them to sketch out an idealized version of any deliverables that your team needs to build.
Ensure that the team members who need to act on the feedback are in the meeting.
Ensure that their feedback is fully and readily digested and actioned before the next meeting.
Give the advisor a chance to respond to the work. Let them roll up their sleeves and make concrete suggestions for improvement (using cloud collaboration tools is helpful here. E.g. Google docs, Figma, etc)
Rinse-and-repeat

Evaluating advisors...

Your advisors should be helping you…

Save money by avoiding blown-out timelines and/or working on the wrong things in the wrong order.
Save opportunity cost by accelerating speed to market (avoiding disruption from competitors and other changing industry conditions).
Increase scale, growth, and profit by increasing the quality and ambition of execution so that the final result is more beautiful, useable, useful, and scalable.

Important: You are not paying for an advisor's time. You are paying for their decades of experience brought to bear in typically short, action-packed conversations that sharpen your focus, accelerate your business, and maximize your outcomes.