作者是ASP. NET QA团队的人, 他和我们分享了他们的测试经验
以前旧的时代 (.NET 1.0, 1.1, 2.0)
当作者加入时, .NET 1.1正要release, 而团队正要开始进行2.0版. 那时候QA的流程是这样的:
1. 当QA team结束对1.1版的测试, 并且签字画押可以发行时, 下一版本正在顺利开发中. 这时候通常有一个"stabilization"的时期, QA team 正在撰写自动化测试, 准备一堆检查清单(我们称做"Exit Criteria"), 以让他们跑在各式各样的configurations上. 也就是说, Dev/PM正忙于做下一版的prototyping和design feature, 而QA正全力于测试目前这个release
2. QA team会根据兴趣和了解程度, 将features指派给团队成员. 我们称为"owning" a feature. 作者一开始时, 是分配到新的GridView and Login controls, spec已经大多数完成, 这些程序已经check-in可以产生build了
3. 一开始要做的事情是先撰写测试计划, 这意思是定义, 你总共计划要做的测试, 及要自动化的项目. 这计划是十分详尽的, 所以页数是蛮多页的. 我们会把这些测试数据放到内部的工具去(Maddog)
4. 测试计划会被PM, 处理这些功能的RD, 以及相关的QA所检视, 试图去找出遗漏的地方或是值得加以增加的部份
5. 接下来是主要测试的阶段, 会分成"ad-hoc"(任意尝试各种各样的scenario, 以发现缺陷) 和撰写自动化测试两个部份
6. 我们会纪录是否我们已经达成测试计划中, 所要求的自动化测试个案的数目 . 例如, 测试计划中定义200个测试个案, 我们会报告还有多少个还没自动化, 还需要多少时间去完成
7. 有一点要注意的是, 当时的情况RD不写任何单元测试
8. 在某些时间点上, 团队已经开始启动运行. 这意味着所有测试以跑在各式各样不同的platforms和configurations, 来找出所有可能的错误. 这时候QA的时间可能主要在处理以下事情: (1) 撰写测试自动化 (2) 分析每次run失败的原因
9. 这时候我们进入我之前提过的"stabilization" time, 这时候会建制一些环境, 来跑各种SKU, languages, X32, X64, stress和performance等等
10. 在几个月后, 我们已经完成所有测试的自动化, 并且跑完所有的测试. 任何发现的错误可能被修复或试之后再修, 但都已经被记录下来. 这时候每个功能的被画押, 产品准备出货. (每次beta relase都会重复同样的流程)
这个流程所产生的问题
1. 过度自动化
- 过于将重心放在自动化上, 测试人员会花费很多的时间, 在编写和调试大量的回归测试, 而不是找出更多产品可能被使用的scenario
- 会不遗余力地, 去自动化你所能想到的任何一切东西
2. 改变所带来的巨大代价
- ASP.NET is a framework, and frameworks go through many revisions and changes before they are released. However, our QA process was not designed to accommodate this. The QA methodology consistently tried to lock down details too early.
- We spent a lot of time writing fully detailed test plans based on the spec for things that were bound to change a lot, and immediately after test plan creation we launched ourselves into an automation frenzy.
- What ended up happening is that changes in the design of a feature where disastrous, the cost of updating all test plans and all the previously written automation was very high.
3. 没有单元测试
- The QA team was responsible for testing everything, from passing every possible value to each public method of a class to end to end integration tests. This amounted to a TON of tests written by QA.
- Since the only automated mechanism to find regressions was the QA tests it was common for the builds to be broken regularly.
4. QA team 太晚找到问题
- A side effect of our automation story was that scenarios were not verified until it came time to automate the test, which usually meant months after the feature was coded.
- All this miss placed effort meant that bugs where found later rather than sooner.
5. QA team 没有像RD/PM一样同时开始加入项目
- In some ways this fact set us up to be the long pole of the team. There were occasions where the Dev had completed all his/her work, but the QA team still had months and months of work to go through.
- It was also common practice for the QA members to have little input on the design of the product. By the time we became fully engaged, most of the features were already designed and implemented.
