Identify what software is to be tested and what the critical areas are, such as:
Delivery of a third party product.
New version of interfacing software.
Ability to use and understand a new package/tool, etc.
Extremely complex functions.
Modifications to components with a past history of failure.
Poorly documented modules or change requests.
There are some inherent software risks such as complexity; these need to be identified.
Safety.
Multiple interfaces.
Impacts on Client.
Government regulations and rules.
Another key area of risk is a misunderstanding of the original requirements. This can occur at the management, user and developer levels. Be aware of vague or unclear requirements and requirements that cannot be tested.
The past history of defects (bugs) discovered during Unit testing will help identify potential areas within the software that are risky. If the unit testing discovered a large number of defects or a tendency towards defects in a particular area of the software, this is an indication of potential future problems. It is the nature of defects to cluster and clump together. If it was defect ridden earlier, it will most likely continue to be defect prone.
One good approach to define where the risks are is to have several [brainstorming] sessions.
Start with ideas, such as, what worries me about this project/application.