What is automated testing?
Automated testing is a script running the test and comparing what happened to what was expected. Somebody writes the steps once in code, a tool runs them on demand, and the result is a pass or a fail with no human in the loop. The collection of those scripts is the test suite.
A test run is unremarkable to watch. A developer pushes a change, a build server picks it up, and a few hundred or a few thousand tests execute in parallel. Ten minutes later the developer sees which ones failed and on which line. That feedback loop is the product, and the tests themselves are only the mechanism.
Suites are layered, and the layers behave differently. Unit tests check one function in isolation, run in seconds and almost never break by accident. API tests check that services answer correctly, and they are stable because interfaces change less often than screens. End-to-end tests drive the real interface the way a user would, catch the most realistic problems, and break the most often. Selenium, Cypress, Playwright and Appium are the tools most teams meet at that top layer. Which one suits is a separate question, covered in our top testing tools for ensuring quality software.
What automated testing is good at
Repetition, which is the whole case: once a test exists, running it again costs almost nothing. The value of a test is roughly how many times it will run, which is why the same script can be excellent in one team and a waste in another.
Regression testing: checking that the last change did not break something that used to work is the highest-return automation in almost every product, because that check has to happen on every release forever.
Scale: a suite runs thousands of cases across several browsers and devices at once, overnight, without anybody staying late.
Speed of feedback: a failing test that reports in eight minutes gets fixed by the person who caused it, while the change is still in their head. The same defect found three weeks later is a different and more expensive problem.
Consistency: the suite runs the same steps in the same order every time, which is exactly what you want from a release gate and exactly what a tired person cannot promise.
Things a person physically cannot do: simulating five thousand simultaneous users is not a manual task at all.
What automated testing costs after it is built
This is the section most comparisons skip, and it is the one that decides whether a suite is still running a year from now.
Maintenance when the interface changes: every renamed field, moved button or redesigned page breaks the tests that touch it. On an active product this is not an occasional event, it is a steady weekly load, and end-to-end tests carry nearly all of it.
Flaky tests: a flaky test passes and fails on the same code, usually over timing or test data. They are the most corrosive problem in automation, because once a suite fails at random people stop reading the results. A red build that everybody ignores is worse than no build check at all.
Test environments and test data: automated tests need somewhere to run and something to run against. Keeping that environment close to production, and the data in a usable state, is ongoing work that rarely appears in the original estimate.
Licences and infrastructure: tooling, device farms and the machines that run the suite all cost money every month, whether or not the suite is catching anything.
Somebody has to own it: a suite without a named owner decays. Tests get skipped rather than fixed, coverage quietly falls, and eighteen months later nobody is sure what it still checks.
None of this is an argument against automation. It is the reason the decision is per test rather than per project. A test that will run twice a year does not repay any of the costs above. A test that runs on every commit repays them in weeks.
The difference between manual and automation testing, side by side
The table below is the short version, and it is worth reading the last two rows first, because the blind spot and the breaking point are what people find out later.
|
Manual testing
|
Automated testing
|
|---|
|
Who runs it
|
A person
|
A script, on a machine
|
|
Time to first result
|
Minutes
|
Days or weeks to build
|
|
Cost of the hundredth run
|
The same as the first
|
Close to nothing
|
|
Up-front cost
|
Low
|
High: tooling, framework, skills
|
|
Ongoing cost
|
Tester time, every run
|
Maintenance, flaky tests, environments, licences
|
|
Best at
|
Judgement, new features, usability
|
Regression, scale, speed of feedback
|
|
Blind spot
|
Repetition and coverage
|
Anything nobody thought to check
|
|
What breaks it
|
Fatigue and time
|
A changed interface
|
The line that actually separates them
Automated testing checks what you already know to check. Manual testing finds what you did not.
That is the whole difference, and every row above follows from it. A suite is a written record of everything the team has already thought of, executed faultlessly and forever. It will never tell you the checkout flow is confusing, because nobody ever wrote is this confusing as an expected value. A person will tell you that in the first thirty seconds and will never run the regression pass two thousand times without missing one.
This is why the framing which is better has no answer, and why which for this test has a clear one every time.
When to use automated testing
Automate a test when it will run many times against something that is not changing much. Everything below is a version of that sentence.
The two questions that decide it
How many times will this test run? Not how important it is, not how complex it is, but how many times. A test that runs on every commit will execute several thousand times a year. A test that runs before each quarterly release will execute four times. The first repays a week of work in a month, and the second never repays it.
How stable is the thing it touches? A checkout flow that has not changed in a year is stable, while a screen being redesigned this sprint is not. Automation is bound to the interface it drives, so instability turns directly into maintenance.
Both high is the easy case, and that is most regression testing. Both low is also easy: leave it with a person. The interesting cases are the mixed ones, and the rule there is to wait. A test that will run often against something still moving becomes a good candidate the moment the moving stops, and automating it early means writing it twice.
Regression testing, smoke tests and the release gate
Regression testing is the strongest case for automation in almost every team. The work is checking that what used to work still works, it has to happen on every release, and the expected results barely change. It is also the work manual testers dislike most, which is a real benefit and not a soft one.
A smoke test is the small suite that runs first and answers one question: is this build worth testing at all. Twenty or thirty checks covering login, the main journey and anything that would make the rest pointless. It should run in under five minutes, and it is usually the best first automation a team writes.
The release gate is the set of tests that must pass before anything ships. Keep it small and keep it stable. A gate that fails at random gets bypassed within a fortnight, and after that it is decoration. If you have no testing process to hang this on yet, start with our guide to building a software testing plan.
Load, performance and cross-browser testing
Some tests are not manual tasks in any form. Nobody can simulate five thousand concurrent users, so performance testing and load testing are automated by definition rather than by choice.
Cross-browser and cross-device testing is the same argument in a different shape. Checking one journey by hand takes ten minutes; checking it across six browsers and four devices takes most of a day, every release. A suite does it in parallel while nobody is watching.
Both of these are worth automating even at modest run frequency, because the manual alternative is not slow, it is impossible.
Automated testing in a CI/CD pipeline
A CI/CD pipeline builds and deploys the application automatically, and automated tests are what make that safe. Without them, continuous deployment is continuous hoping.
The arrangement most teams settle on has three tiers. Unit and API tests run on every commit and must finish in minutes. A fuller suite runs on merge to the main branch. The slow end-to-end and cross-browser tests run nightly, because they are too slow to sit in front of a developer waiting for a build.
The value here is the speed of the loop, not the tests. A defect caught eight minutes after it is written is a trivial fix. The same defect found in a release candidate three weeks later involves several people, a branch nobody wants to touch and a difficult conversation about the date. That gap is the real return on automation, and it is the one number nobody can put a figure on for you.
