Functional testing of your Mendix application while applying an agile way of working with short sprints comes with the necessary challenges. Together with the release of Mendix 7, we will also release the Mendix Application Test Suite (ATS), powered by Mansystems. ATS will help you with those challenges by allowing you to easily create and record test scripts for your Mendix application that can be run automatically.
In this webinar we will show you how ATS can help you get a better grip on testing your Mendix apps.
Friday, September 30, 2016
10am EDT | 4pm CEST | 3pm BST
View past Expert Webinars on-demand at developers.mendix.com.
Download slides: Mendix ATS Webinar
How do you deal with a test driven development (TDD) approach?
With ATS, you set-up test cases and test steps before the start of your development cycle, and use these test cases as guidelines in your development process. These tests cannot be executed before you start modelling.
Are there plans for Selenium WebDriver Java integration with ATS, or will all scripts need to be recorded?
Yes, ATS is integrated with Selenium Web driver. Tests are interpreted at runtime and sent to Selenium. Test steps can either be recorded or manually set up via the ATS user interface.
Which Mendix versions are supported by ATS?
ATS works on Mendix version 5.14 and up.
How do you specify which code version to hit in test?
ATS works against a running version of your app. Therefore, it depends on which version of your model you have deployed.
How long does it take to implement ATS?
ATS is installed and configured within a few hours. The training of the tool takes a few days. As with all software, organizing the governance of the software in your organization typically takes more time.
What would occur if a test fails, and how would you see the reason for failure?
ATS shows whether a test case failed, at exactly which test step it failed (including an error description), and (depending on your Selenium provider) provides a screenshot and/or movie of the moment it failed.
Is ATS limited to Mendix applications? If you have another web application that calls the Mendix application with some parameters, and Mendix sends some parameters back, could that be included in the test scripts in ATS?
ATS is not limited to Mendix, but has been designed for Mendix. ATS supports widgets provided with Mendix out of the box. If you would like to use it for different platforms, you need to build your own way of identifying page elements on such platforms. This requires custom work, which we can help with.
Seeing that ATS indicates set execution time, could one delay some inputs or actions to simulate real user experience and then run multiple scripts at the same time to simulate a performance test?
ATS can execute multiple tests parallel; however, it has not been designed for load or performance testing. You can also add a static wait. ATS focuses on functional testing.
What is the licensing structure for using ATS?
You need an ATS license for each Mendix app you want to test. You can also obtain one bundled license to test all your Mendix apps.
Can ATS perform series of tests where complete test sequences are executed on various browser brands and versions (in order to check if the application works correctly under various browsers)?
Yes, ATS performs series of tests by using tests suites or data-driven tests. You can select between Chrome (versions 51-53) and Firefox (versions 38/45 ESR, 44, 45) browsers. All tests run in supported browsers without modifications.
Can ATS verify the contents of generated files by the Mendix application, for example and 'export to XLS' button, by comparing the contents with reference files?
The interpretation of a downloaded file in a test case is currently not in scope.
How much database space is typically necessary for an ATS test script and its test results?
The required storage for test cases is low. To keep control over the number of logs we will introduce a scheduled cleanup mechanism.
Can ATS check if a certain GUI item is present or absent, for example a button or menu item, for the current user?
Yes, ATS can be used for positive and negative tests to check whether elements are visible or invisible.
Are there possibilities in ATS to speed up the execution time of a test compared to the time it was necessary to record the test?
The execution time of test cases is usually "as fast as possible". The timing during recording does not affect the execution time.
Are there possibilities in ATS to speed up the execution time of a complete suite of tests by running them in parallel?
Yes, you can speed up test execution by running them in parallel.
Can ATS export the results (Pass/Failed) of the tests to an external file (for example XLS/CSV)?
Yes, ATS supports exporting test results to PDF.
Can we perform data-driven testing using ATS?
Yes, you can upload your own data sets into ATS and drive your test scripts using that data.
Can we view recorded script in ATS?
Yes, all your test scripts are kept in ATS and are available for use, viewing, and modification.
Rather than duplicating a part of the test script, can we make it into a reusable method and parameterize it as per our needs?
Yes, you can modularize a part of your test script into a custom action and reuse that in a flexible way by introducing parameters.
Is there a way for failed ATS scripts to automatically generate a bug in the Mendix Support Portal?
That functionality is on the roadmap, but not yet available.
Will this be available for on-site installations?
Yes, but your application needs to be accessible for the Selenium provider ATS uses. To achieve this, some changes to the firewall may be necessary. We can assist we setting this up.
Does ATS have the ability to insert conditional checks at any point in the test script?
Examples of conditional actions:
Yes, ATS can do conditional checks at any point in a test case. You can do positive and negative assertions to check whether something is (or is not) visible or has (or doesn’t have) a certain value. You can also add a static wait. Implicit waits are applied automatically.
Does ATS have ability to invoke/insert actions in the test?
Can ATS run tests in the background so that testing doesn't take control of QA/QC user's machine to run tests? Can ATS run different tests at the same time? Can ATS run tests for a specific number of times or for a specified duration of time?
All tests in ATS are controlled by the ATS server and run on a Selenium server instance. It is not necessary to take control over a user's machine. Multiple tests can be run simultaneously. Scheduling is possible to trigger (periodical) test runs automatically at certain times.
Can ATS handle errors in test scripts? For example: stop test if error, or continue test if error. Configurable Error handling notification - system notifies of errors by taking screenshot of the error and/or by sending a notification via email (test name, test step, test time etc.).
By default, a test script execution stops when there is an error. For a proper clean up (removal of data, logout, etc.) it is possible to define a teardown section in your test case. On failure of a test step the execution proceeds with the teardown section. When there is an error, a screenshot is created and saved with the log.
Can ATS save test results to a desired format (xlsx, pdf, docx, txt)? Is there a mechanism to tell if a test passed or failed without having to look into the details of each test’s results?
Exporting test results to PDF is supported. The results screen gives a clear overview of the results of the executed test cases.
Does ATS have a Scheduling manager for running tests?
Examples: daily, weekly, monthly, once, or run a test case multiple times.
Yes, the scheduling manager in ATS allows flexible automatic execution of tests scripts regularly (daily, weekly, monthly) or one-time only.
Can ATS send email notifications?
A notification mechanism for ATS is planned. You can check all latest test results in a glance with the new project dashboard.
Is ATS available for free applications?
ATS is currently not available for free applications.
Is the test (Selenium instance) executed on the Mendix cloud?
We will not be running selenium in the Mendix cloud.
When we create a new page or button, can we add test cases to test these items within the modeler?
Running tests involves more than running Selenium. Providers like Sauce labs and Browser Stack also provide images with correct operating systems and browser combinations, and in some cases mobile devices to run tests on. We will use these services to provide high quality test execution environments.
Will there be any versioning, date, or execution time provided in the report for the test suite? And will the report drill down to test cases too?
The reporting functionality shows the executed test cases, execution time, and results per test case (drill-down available to the step level).
What does ATS use for identifiers?
ATS makes use of the "mx-name-<widget name>" CSS class names generated by the platform and assigned to the widgets in your app. You modify the widget name in the widget’s properties in the modeler.
Is there a way to edit your script in a Selenium/Java format, or is the IDE style way the only way?
Test cases are not compiled into code but interpreted at runtime. Therefore, they can only be edited in ATS.
Could you also create a test script where you select a specific template grid item, for instance open the 'webinar expenses' report, and add an expense?
Yes, this is possible. You can target template grid items either by their index (first, second, …) or by their content. E.g. find the item that contains a certain label.