The situation is that you have a medium to large scale project you need developed. You don't have an in-house development team or one that is available. Additionally, creating a new team is not an option. Basically, you need to go outside of your company to get this project done.
Here are five quick tips to help you effectively screen vendors.
1. Consider Multiple Vendors
Unless you have experience with a particular vendor, look at several. This is no different then hiring an employee. Make sure the vendor can understand your business needs and can speak intelligently to both the technology and the business domain. Consider their core competencies, certification and experience. Insist on speaking to past customer references.
2. Technology Considerations
This depends. If you actually don't know much about technology, platforms and so forth, you will not be able to determine whether a vendor's recommended technology is based on the vendor's preference, or based on your business needs. There are a few things you can do here.
a). Require vendors to justify their technology recommendations in their proposals.
b). Hire an independent resource to help you evaluate vendor recommendations or to perform a technology review based on your needs.
c). Beware of technology bias (unless you are already committed to a specific technology). Many smaller vendors focus on a specific technology like Microsoft or Java, etc. On the the other hand, vendors who focus on a specific technology are often extremely competent in that technology - consider this.
Here are some comments a vendor might say that exposes his/her technology bias:
"We develop in XYZ because we've determined that it is better than anything else on the market."
"Everybody is using this technology."
"This technology is the wave of the future."
Here is an honest statement from a vendor:
"I don't know squat about ABC technology and whether or not it will meet your needs, but I am an expert in XYZ and have determined that it will meet your needs."
Developers are developers because they love technology and they especially love the challenge of those on the bleeding edge. RUN AWAY FAST from any vendor who recommends using the latest technology, methodology, development platform, etc. The last thing you need is to have your project become the test bed for some new, unproven technology. Remember, new technologies demo impressively, don't be lulled into early adopting.
3. 100% Developer Commitment
If you are providing full-time work, make sure the vendor agrees to a named developer or more to work exclusively on your project for its duration. It is not unreasonable for the vendor to require that you agree to a minimum number of hours per week per developer.
CAUTION: It is not uncommon for smaller vendors to take on Out-Tasking projects (small low-effort projects and development tasks). If you are providing full-time work, it is fair that the vendor grant you exclusive use of his/her development resource. Furthermore, a vendor who needlessly overworks their staff (excessive hours beyond full time) is a slave-driver and doesn't deserve your business.
4. Phased Approach and Status Reports
Make sure the vendor agrees to a phased approach and weekly status reports. Run away from any proposal that doesn't give you visibility into the project (the blackbox approach). You want to make sure that that you have check points throughout the development lifecycle. This is a good point to bring up in an independent review (see below).
HINT: An experienced vendor will assume this. Ask about the development approach they use. If they don't offer this, you'll need to ask. While this is not a disqualifier, it is a red flag.
5. Independent Review
Have the vendor agree to an independent review of their work at points during the development process.
An honest vendor will not only agree to this, but will consider it good for the overall project. I have been on both sides here and in every case, it resulted in further insight that only improved the overall project.
The process you use to select a development team is as important as the development process itself. It can be the factor that makes or breaks your project's success.