What's New in ATS 2.0

What versions of Mendix is this supported on?

We officially support Mendix version 5.14 and upwards. For more information, see the Mendix Platform Compatibility.

What license do I need, and how many independent licenses do I need?

You need both an ATS License and a license for a Selenium solution such as Saucelabs or Browserstack. Instead of using a hosted service, you can also create your own Selenium server.

Is it possible to get a test account?

We can provide temporary trial access to ATS for evaluation purposes. Contact your account manager or customer success manager for more details.

How many simultaneous testing sessions can you create at one time? Could this replace penetration testing?

The number of parallel sessions is determined by the Selenium service that you use. For instance, a trial subscription at Browerstack supports up to five parallel sessions, but you can increase this number to some extent by choosing another subscription type. As for penetration testing, it is not possible to use ATS for these kinds of tests as it is not what it's designed for, but there are very good tools available that have specialized functionalities for penetration testing.

Instead of using a data-set to fill field values, is it possible to use random values or value ranges as well? For example, between 1-1000?

Yes, this is perfectly possible. There are actions available called Random String and Random Number, that generate random strings with specifiable pre- or post-fixes and length, or random numbers within a range of choice.

Can test cases be and exported to Microsoft Excel from ATS?

Importing test cases from or exporting to Microsoft Excel is not  possible at the moment. You can import and export test cases in XML format to move them between projects, or export test cases (definitions) to PDF files to show a detailed description of each test step. Importing from and exporting to Microsoft Excel is currently limited to test data. For more information, see the documentation on Data-Driven Testing.

To use this automated test with a bigger load, we just need to create a bigger data-set right?

In short, ATS would not be the tool of choice to do a load testing with a reasonably high load. In principle, you could do a load test by creating a large data-set and running it in parallel, but this is limited by the number of parallel sessions your Selenium server supports. For instance, a trial subscription at Browserstack supports up to five parallel sessions, but you can increase this number to some extent by choosing another subscription type. However, load testing typically uses far larger amounts of concurrent sessions than you could create in this way. We are currently working on a new version of APM, APM 2, which will support load testing, so keep an eye out for that!

Can the tool handle setting drop down values?

Yes it can. This is also shown in the demo part of the webinar, where we recorded filling in a form. You can choose a value from a drop down menu, or you can do this by manually adding a test step.

How about integrations and unit-testing?

At the moment, since ATS is based on Selenium, ATS is able to do all kinds of testing, as long as it is in the application interface. This means that integration and/or unit testing are not supported right now, except by a workaround where you create a page or pages with buttons triggering microflows representing these tests.

Are there any future plans to support integrations from ATS, for example, using another platform instead of Selenium?

There are no plans to replace Selenium, which is the de-facto standard for browser-based testing of things that are not browser based, such as API and/or unit testing. We do plan to support API testing in the future, but this will not replace Selenium as the platform for our browser-based tests.

Were any mocks used to prevent tests failing when API calls fail?

In this webinar and demo, no mocks were used. In principle, there is no problem testing with ATS when you are using mocks, as long as the test steps can be executed from the application interface.

What technique do you use on Selenium?

ATS uses the web driver API to instruct Selenium servers to execute your test actions in a browser. For more details about the web driver API, see Getting Started with Selenium for Automated Website Testing. Instead of having to manually code the web driver instructions, a Google Chrome plugin is used to determine what a user is doing in a browser.

Regarding the drop down option selection: What is the best way to make sure drop down selection is dynamic, and assuming there is a heavy retrieve action behind the drop down, will the script be able to always find the right selection?

Assuming that the question concerns drop down menus that use data retrieves to determine the options that will be shown: the function that selects a value in a drop down menu has an internal mechanism that waits for microflows and data retrieves to finish. Therefore, ATS should be able to find the right selection.

Is it possible to run tests on mobile devices instead of using a web browser in order test apps that are predominantly developed for mobile usage and natively deployed as an APK or IPA?

At the moment, this is not possible. However, we are currently working on adding support for mobile testing to ATS, which will be released gradually.

How is ATS connected with APM when adding a load of, for example, 5 concurrent users? Is there any documentation on pricing, Mendix version support and other such topics?

ATS and APM are two separate tools. However, you can use ATS to execute test cases and simultaneously monitor the performance with APM. We are currently working on a new version of APM, APM 2, which will support load testing, so it might be interesting to keep an eye out for that. We officially support Mendix version 5.14 and upwards, for more information see Mendix Platform Compatibility. Contact your account manager or customer success manager for pricing details for Add-ons, single app options, more app users, and cloud resources.