Quality assurance is like cleaning a bathroom; you may not want to do it but it needs to happen.
QA is about ensuring something works and ensuring it can’t be broken. The second part is far more work that the first. Testing if something works is easiest. If the program has an interface then a user can sit down and use it. If they can enter new information and it saves properly and they can view information then it works.
User interfaces are the easiest to test. They begin with a visual check. Is the interface simple and easy to use? Can you tell from a glance what it does and how to use it? Simple feedback like this will be very helpful to programmers because simple changes can make the interface far more usable. You must be extremely detail oriented. Are form controls consistent? Do they have the same style? Are there any inconsistencies? A QA specialist will notice when individual pixels are off.
Accessibility tests are next. Take a screenshot of the form. Use an image editor to change it to black and white. If the interface remains clear and readable then it will probably be clear for any form of color blindness. If the form has messages then ensure they don’t use color coding. The messages should clearly say what they are, whether it is an error, warning, info or news. Some programmers or designers try to be clever with colors but that cleverness tends to fail for color blindness. After color checks comes input accessibility checks. Some people can only use a keyboard so input forms must be usable by a keyboard. (Except some rare controls that can only be used with a mouse.) For phone apps the opposite is true. A phone app should use touch controls for as much as possible and keyboard only when absolutely necessary. Next look at the accessibility options of the system. If the user has larger fonts will that affect page layout? If they use the zoom option does the interface remain usable? All these checks can’t be automated. A person has to perform each test. There may be other tests too, but these depend on the platform.
User interface function tests ensure the interface functions properly. Ensure it displays data properly and new data can be entered. Some interfaces contain multiple modes. A form that can view the data may also allow editing, adding or deleting. Create a state diagram for these modes. Ensure that the states transition properly and that data is saved/cleared/displayed cleanly. Most forms have validation, so check every field to ensure validation shows. Ensure the form controls are large enough to display the contained information.
The next task is to break it. A QA expert can break software easily by entering unexpected things. Programmers need to learn QA for this reason, so they learn how to break things. Once they learn how to break something they can program so it can’t be broken. Enter unusual characters, Unicode, formatting characters and anything you can think of. Give someone an age in the future, or an age a thousand years in the past. Ensure all input is validated. Most interfaces these days are web based and that leaves something else to test. Send data directly to the web server. You would be surprised at how many servers assume pre-validated data. This is true for any client-server application, including any online games.
QA is a lot of work. If you release many small versions of a product then it has to be done for each release. The cost is often initially ignored, but as the project grows QA can quickly eat up more cost than programming. Use automated tests to reduce this cost. An automated test utilizes programming instructions to mimic the tests done manually. People have used all kinds of methods to create automated tests, including keyboard and mouse click replication then checking images. While this works, it is extremely tedious. It’s far better to create an API for the interface. API means application program interface. It lets code work directly and easily with the interface. Test cases are coded in a script and they use the actual interface for testing. Automated tests can’t check visual aspects, like a human eye and mind, but they can check for function and break tests. A good rule of thumb is that automated tests take as long to make as writing the program.
Testing isn’t limited to visual interfaces. Programs also have functions, classes and APIs. Each should be tested. Luckily automated tests work really well for these. You give each function or class different data and it provides the correct output. When states are involved, every state and transition has to be tested. Sometimes unexpected bugs crop up. When bugs are found, create a test for it first then fix it in the code. This way as problems are found, the automated tests grow more comprehensive.
Automated tests open the door to new types of tests. A Monte Carlo test sends random data to a function or interface then checks for success or failure. It’s a good way to find cases you didn’t consider. Running the same set of tests over and over will quickly reveal memory leaks or threading issues. A stress test is when you run the system to the max to see how it performs. Some software will eat up file descriptors or memory. Memory is especially important because it may put the operating system into intensive swap, which effectively takes down the server. Stress tests also quickly reveal threading issues. Finally automated tests are good for performance tests and profiling. This can be used along with profiling tools to determine which parts of your program take too long to run. This can reveal algorithm problems.
The final step in QA is hacking. Everything up until this point has been testing by following the rules. When you hack something, you break the rules. Examine the protocols used and try to figure out how to break them. If it’s a browser based app then see if you can mess with the cookies. Look at the network traffic. If something looks odd then try to twist it into doing something it wasn’t meant to do. Edit local data files with another program. If it’s an application, and that application is meant to be secure, then hack it. There are plenty of tools to help with this. They let you search and edit the memory of another application or change its code while it runs.
QA is a skill separate but linked to programming. All great programmers have great QA skills because QA provides feedback that improves programming skill. QA is also required for a great product. A QA department is also essential, but it should be treated as beta testers. Ideally, a QA department should receive a product with all automated tests written and passed. The product should mostly work. The QA department should find things that developers happened to miss. This includes problems related to different kinds of computer hardware.
Some software development approaches try to get rid of dedicated QA. Agile is popular and attempts to do this. Agile is murky, meaning there are many flavors and everyone has their own favorite. The ideal is to start with automated tests then add code until the tests all work. In reality, programmers just sit down and type out code and try to make everything work. Sometimes, QA is even left to the client. This invariably leads to many bugs left undetected, unless you have really good programmers.
Open Source projects also avoid the whole notion of QA, but like Agile it’s murky. Some projects are well written with complete tests but most rely on users to report bugs. The inevitable result is that common features work fairly well while features rarely used often don’t. If a project continues for several decades then the result can be good. Unfortunately, a poorly working project is rarely used which means it rarely lasts that long. There are exceptions to this, of course, but the majority of Open Source lacks QA.