Testing

Testing is a primary component of any standard Software Development Life Cycle (SDLC), and companies often spend millions developing these standards. Following the purchase and start of any OneStream implementation (or enhancements), one of the earliest conversations that needs to happen is how to test the software post-build. This chapter will discuss various definitions of testing and when to use these testing methods, plus some of OneStream’s recommended testing practices.

OneStream is committed to the success of all implementations and the success of administrators following implementation. This chapter aims to guide how to test various enhancements found later in this book (and even outside the scope of this book).

Armed with the definitions – and when to use the proper testing method – the administrator can make requested enhancements or changes for any organization. They will be able to test and roll out enhancements properly to satisfy any internal or external audit requirements and, most importantly, provide absolute comfort to the organization that enhancements or modifications will work properly.

Types of Testing

There are many types of testing, and this chapter is not intended to cover every possible testing type or testing scenario. This chapter is intended to capture the most commonly used testing methods, provide a brief descriptive definition, and the best use case for employing the testing type.

The most common types of testing found during a OneStream implementation are:

• Unit Testing

• Integration Testing

• Data Integrity Testing

• Data Validation

• User Acceptance Testing

• Smoke Testing

• Performance Testing

• Regression Testing

• Parallel Testing

Unit Testing

Unit testing is defined as individual component testing, and a tester can utilize very targeted unit testing or component testing. The unit tester should be the person or the team that built the original artifact. One of the main reasons to perform unit testing is to ensure that the artifact is working prior to being reviewed by the customer or end-users. One example of such targeted testing might be validating that the import step is working under the Actuals workflow.

In turn, an example of component unit testing might be where we validate that the user can import data, create a journal, post a journal, and enter data in forms under the Actuals workflow parent.

Both examples illustrate unit testing; however, one is more granular than the other. One specifically targets a very specific step in a workflow, whilst the other targets all the steps under a parent workflow. Whatever the case, though, they are both confined to the workflow section of the application.

Unit testing is commonly seen with business rule writing, where the rule writer will need to find a way to test the output of the business rule. If the writer is testing a seeding rule, they might create an Excel spreadsheet using OneStream’s Excel add-in tool. They select the dimensions from which they are copying (from within the application) and then create another column for the targeted location of the seeding rule. The business rule writer can then use Excel math to determine if the seeding rule is working properly and if the writer is selecting all the desired or correct intersections between the source and target values.

Integration Testing

Integration testing is when application components are tested as a group; it is typically conducted after unit testing. As with unit testing, integration testing should be performed by the person or group that built the original artifact to be tested. Integration testing would also be commonly used to test the movement of data within an application or into the application and applied to test the import of data into OneStream via a new connector or flat file upload. It could also be used to test the movement of data from Stage to cube. These are all types of integration testing.

Picture the following. You have tested several individual pieces of OneStream, but you want to test whether the individual components work together. Using the same parent workflow unit testing example referenced above, we will expand this to an integration test example.

We have imported our data, created a journal, posted the journal, and entered forms. Now, we need to validate that everything is consolidating and aggregating properly. This is a great example of integration testing. Integration testing can be expanded even further from this one example. We can layer security validation as well. One can now see the importance of integration testing and how it can be applied.

One other example of integration testing might follow the original implementation. Let’s assume the company needs to implement OneStream’s People Planning solution. Here, the integration test will check the new connection is receiving data into the People Planning register. In contrast, another integration test will verify the movement of data between the register and the cube or alternate tables.

Data Integrity Testing

Data integrity testing follows the lifecycle of data as it is imported into OneStream to the very top of the consolidation/aggregation. Data integrity testing is typically performed by the person who has built the artifact. This type of testing, however, often works in conjunction with the customer. The company building the artifacts knows how to build, but they may not have the detailed data knowledge that the customer would have. This type of testing becomes critical when a customer has a complex ownership structure or possible elimination issues. For example, if you own Subsidiary A at 100% and Subsidiary B at 50%, you expect only half of Subsidiary B’s values to consolidate to the top.

Data integrity testing is similar to integration testing; however, the tester will validate that the end number is the correct resulting number. Integration testing, by comparison, only cares that the data is imported or exported correctly within the application.

Data Validation

Data validation is crucial to testing an application. OneStream often becomes the book of record for a company. The customer will always own both the data from the old reporting system and its movement and transformation into OneStream. Implementation partners support the data validation effort, but the customer has the intimate details of the business that generates the data. For that reason, implementation partners can’t own the data validation process. Customers typically assume that the data is clean and organized. However, the implementation often uncovers unique data challenges they hadn’t anticipated. These challenges can lead to project delays or missing deadlines. Data validation will always be the biggest unknown variable in a project.

User Acceptance Testing

Though I will attempt to do this section justice, the OneStream Foundation Handbook does an excellent review of user acceptance testing. (See Chapter 2, Methodology and the Project.) I will offer a slightly different spin, but it is worth reading that section as well for additional information.

User acceptance testing, or UAT as it is more commonly known, is the phase of testing where an intended audience (typically selected members from a company or organization) runs through real test scripts for a desired testing result. This type of testing can be very singular or individualistic, or this testing can have a complete end-to-end outcome.

UAT commonly occurs after the completion of the final build items and usually after some end-user training has been delivered. This allows the end-user to know what they are testing and gain familiarity with system navigation. Users are typically given a set of scripts for testing, allowing the natural Software Development Life Cycle (SDLC) to maintain traceability as to what was officially tested.

UAT scripts can be written to be very granular or high-level. It is commonly left up to the customer to determine at what level they prefer the scripts. The customer typically writes the scripts with support from the implementing partner unless otherwise agreed upon. The customer will know from the auditing partners what the key testing components will be (e.g., security; calculations such as the balance sheet being in balance or bringing current year net income into retained earnings).

OneStream typically recommends not focusing so much on basic, native functionalities, but rather to focus on the more nuanced functions of your testing scripts (important for the auditing teams). For example, you might want to redirect any focus on posting a journal (which is basic and inherited functionality) in order to ensure that a recurring journal is set up properly, which is more nuanced functionality within OneStream. Another example is the adding of a dimension member, which is basic functionality, versus a focus on the member that has a Member Formula, and that the Member Formula is producing the expected output. An example of a very granular type of test script can be found in Figure 2.1. An example of a high-level test script can be found in Figure 2.2.

Figure 2.1

Figure 2.2

Individual UAT Script

During UAT, test scripts might be written to test a very specific component of the system that impacts one user or a group of users only.

An individual UAT script could be something like validating journal creation and posting that it is working with the proper security applied. This type of testing will be very similar to the unit testing mentioned above, but instead will be performed by an actual selected end-user. These types of tests are intended to give comfort to the end-user that the system is working as expected.

End-to-End UAT

End-to-end testing covers an entire process or even the entire monthly close process; detailed coordination, communication, and support are needed to accomplish this type of testing.

OneStream’s example of end-to-end testing would look something like the following example. The system administrator imports data into a workflow and notifies the consolidation team that the data is loaded. The consolidation team then validates the numbers, creates journals, and manually enters key metrics not captured in the import. The consolidation team will notify the manager that the journals are ready for review and ready to post. The manager will then post the journals, validate the numbers, and notify the controller that the workflow is ready to be certified. The controller will then certify the workflow – after the proper validation checks – and notify the system administrator the workflow is ready to be locked.

This type of testing takes a significant coordinated effort but can realize real value as it crosses many teams and touches many hands (much like it would if it were truly a month-end process). When possible, it is always recommended to perform at least one of these types of tests, however burdensome they may be, as they create extensive value for the overall testing process. These tests usually highlight possible process issues or point out system problems that would not be caught during normal individual component testing.

Smoke Testing

Smoke testing is validating – at a much higher level – that the system is operating as expected. Smoke tests normally occur after migration (often called cutover). Smoke testing is typically performed by the customer following artifact migration and may not be applicable during a brand-new implementation. It will definitely be applicable after the initial go-live and the implementation of any additional enhancements. These smoke tests do not normally validate any data but utilize button clicking to ensure nothing is erroring out after the migration from the UAT environment to a production environment. This type of test will allow the implementor to know if a key component has been forgotten during the migration effort.

For a standard OneStream implementation, it is recommended to test all the buttons on a dashboard. This is typically where a smoke test becomes valuable. As an example, I have just migrated my artifacts from UAT to production and am tasked with smoke testing. I receive an error when I push a button. After investigating, I realize that only half of my dashboard components were migrated, and I am able to quickly correct that before the system goes live.

This type of sanity check is very important but not meant to capture every aspect of the testing cycles performed in development or test environments.

Performance Testing

Performance testing is the process of attempting to break the system by running so many jobs or processes in parallel that they overload the physical server’s capacity. Much like the end-to-end UAT process, a performance test can often take a coordinated effort. Performance testing typically gives companies the comfort that the hardware purchased – in conjunction with the design and configuration that supports the OneStream environment – is sufficient to support the end-user experience.

Process-Oriented Performance Testing

Process-oriented performance testing runs as many processes as possible – at once – to see how the server is performing. This testing takes a coordinated effort to employ multiple users to click a specific button, which kicks off an intense business rule to load data, consolidate data, and action other tasks at one time. The administrator will typically monitor a diagnostic tool or validate the results afterward.

This type of testing could be automated in some way by creating a business rule to run data management sequences. This type of business rule could limit the number of end-users required to perform this type of testing.

User Load Testing

User load performance-based testing maximizes the number of concurrent users utilizing the system and then adds additional users to perform additional tasks. This isn’t necessarily about overloading the system with many processes but testing the end-user experience for latency or lag. Is the end-user having an optimal experience in terms of system performance? Are reports rendering properly in a quick timeframe?

To provide a concrete example of this type of testing, let’s assume the GolfStream company (OneStream’s demonstration company) is provisioned for 100 software licenses but only 40 concurrent users at any given point in the application lifecycle. It might require the company to coordinate those 40 users effectively and add an additional 20 users to perform ancillary tasks within the system. Under test conditions, the system administrator may monitor diagnostic tools and end-users will report if they experience any latency with rendering or system utilization.

As mentioned above, some of these activities could be replicated by building a business rule to imitate the needs of particular users.

This type of test really gives the entire organization comfort that the system and hardware purchased will be adequate for go-live.

Regression Testing

Regression testing is the revalidation of historical test cases. The most common use for regression testing is during a software patch or update. Typically, after the software is updated, the existing list of scripts from both unit testing and UAT cycles is sufficient for this type of testing. Software updates are not the only occasion for regression testing, though. Often, auditing partners will request these types of tests to be performed annually, and it will solve most of the testing standards around Information Technology General Controls (ITGCs).

Parallel Testing

Different types of software may lead to different definitions of what parallel testing means. OneStream defines parallel testing as running a month-end close process from beginning to end within OneStream and comparing the results between the legacy system and OneStream. This type of testing doesn’t necessarily need to be run concurrently with the other reporting or planning system; a customer can opt to have a parallel on historical periods. However, when possible, it is recommended to do at least one parallel testing cycle with your existing reporting or planning system. This type of testing will give absolute comfort that the system is generating the same results and allow a smoother transition from a legacy system to OneStream.

Standard OneStream Implementation Testing Cycle

Standard is such a broad term, and testing is never a one-size-fits-all approach; many factors affect many of the key decisions as to how a company will enter testing cycles. However, when sizing an implementation, implementation partners often have to make some baseline assumptions unless told otherwise by the customer. For example, implementation partners will assume that there will be a constant need for unit testing, and that testing will live alongside the building of application components. Partners will also assume that the customer is satisfied with one UAT cycle, and that two parallel testing cycles will take place.

There are many factors, however, that might increase testing or (conversely) cut testing to the bare minimum. Some companies want great comfort when going live and spend extra time testing; some companies want to be live as soon as possible and are willing to assume the risk of implementation defects or quick fixes in production. Other companies prefer a methodical testing approach and target the most optimal end-user experience. These companies are willing to push go-live to ensure that testing has flushed out any possible issues that could arise.

Conclusion

Testing is more of an art than a science, even though testing is a pass-or-fail exercise. Naturally, the science part really comes to the fore when the tester can see quantifiable results through the passing or failure of individual steps or scripts.

End-users will often explore the system outside the bounds of script steps when they participate in testing. This is where implementation defects that are not part of specific scripts are found and fixed. Often, companies will fix processes or find that something needs to be tweaked with this testing; this can be frustrating but leads to a better product overall. This is what makes testing an art… allowing end-users the freedom of application exploration.

Another great element of testing and testing cycles is the feeling of accomplishment. When you enter a testing cycle, the implementer is hopeful that everything works on the first try and there are no implementation defects or issues. This is not an attainable goal! There will always be fixes that need to happen in a testing cycle; it takes multiple testing cycles to finally get a clear and clean cycle.

Testing will set the standard for your implementation, your enhancements, and even minor fixes. Testing is often the project phase for the first impression on an end-user. More testing takes time but will be rewarding in delivering an optimal product in the end.