Testing a system that is composed of a variety of concurrent subsystems can be very complex operation. Good planning, a thorough understanding of the behaviour and the state of the system, tools to analyse the flow and the data, and tools to automate the process are all required to achieve automated end to end testing.

This paper outlines what end to end testing is and examines some of the issues.

What is End To End (E2E)Testing?

Many subsystems are tested in isolation. This is fine for many scenarios that are local to that subsystem, but there are some scenarios that cross the boundary of several subsystems. This is good because a lot of testing can be done in isolation but at some point a complete end to end test needs to be carried out on the whole system.

A good example can be found in banking where a salesman will accept a trade, pass the order to a trader, the trade will be executed on an exchange and the result will end up in risk management, settlements and accounts. This is a complex set of transactions typically involving a variety of systems which are difficult and time consuming to test.

In the trading context the end to end test would comprise a trade entering the system at sales and seeing a result in settlements, accounts and risk management. The result should be a consistent state in all the systems involved.

Problems and Issues

Unfortunately a lot of testing of complex systems is carried out manually because the level of effort to automate the E2E test is considered to be too expensive. This is primarily due to the fact that testers take a user interface centric approach to testing and do not have the necessary development skills to produce the tools needed to automate testing. Often they rely on tools such as WinRunner to drive the user interface.

I said manual testing is unfortunate because as we all know systems are not static, they evolve. Sometimes this evolution is minor and does not impact other systems, or so we think. At other times the impact is greater and the E2E tests need to be rerun. At this point we take the hit retesting system with the inevitable delay to delivery. But what if a test fails? Do we restart and rerun all our tests after the fix is made. In most cases this is unlikely. The failed test is simply rerun and the hope is there are no side affects.

This haphazard way of testing may be fine when the risk to life or loss of money is low, but it becomes increasingly difficult to justify when the risks are high. A better approach is to automate end to end testing so quality is guaranteed.

Given that more business are trying to become agile in order to respond quickly to changing business needs the need for automated E2E becomes stronger, for without it testing cycles can significantly delay the introduction of new functionality. I’d go as far as saying you won’t be truly agile until you’ve automated E2E testing.

Objectives

The objective of E2E is to look at the state of the system as well as its behaviour. A realisation needs to be made that the user interface is not what we are primarily interested in as it simply provides a representation of the underlying data and displays some of its behaviour. In E2E we should be more interested in the underlying data and scenarios across all subsystems.

Clearly fully automated E2E testing can be very time consuming and to get the most value a 80:20 perspective is useful, Looking at the scenarios of a typical system only about 20% of the functionality is used on a regular basis. So the initial objective should be to automatically test the 20% first and gradually adding the lesser used functionality.

Requirements

Given that we need to look at the data and the system behaviour and not just the user interface a different set of skills are needed to automate E2E as specialised tools will almost certainly be required to drive the system as well as possible instrumentation of a subsystem to extract the data necessary.

In addition to development skills analytical skills are necessary to determine the 20% of functionality that is really worth testing.

Finally as with all projects, good project management is mandatory. I’d advocate an agile approach to be most successful, that is, short value driven iterations delivering useful tests as quickly as possible to get feedback as early as possible on the effectiveness and potential cost of automating E2E.

How We Can Help

SMR can you help you by navigate through this information and successfully implement E2E by,

• examining your current system and planning a path to E2E,
• making tool recommendations,
• implementing costume tools,
• developing test cases,
• implementing E2E test packs,
• support test packs.