============================================= Automate Browser Testing with Robot Framework ============================================= This document provides details about CumulusCI's integration with `Robot Framework `_ for automating tests using CumulusCI, Salesforce APIs, and Selenium. Why Robot Framework? ==================== Robot Framework provides an abstraction layer for writing automated test scenarios in Python and via text keywords in ``.robot`` files. Since Robot Framework is written in Python (like CumulusCI) and has a robust SeleniumLibrary for automated browser testing, it works well with CumulusCI projects. CumulusCI's integration with Robot Framework allows building automated test scenarios useful to Salesforce projects: * Browser testing with Selenium. * API-only tests interacting with the Salesforce REST, Bulk, and Tooling APIs. * Complex org automation via CumulusCI. * Combinations of all of the above. The ability to create rich, single-file integration tests that interact with CumulusCI's project-specific automation, Salesforce's APIs, and the Salesforce UI in a browser is the most exciting feature of the integration with Robot Framework. Robot Framework makes it easy to automate even complex regression scenarios and tests for edge-case bugs, just by writing Robot Framework test suites and with no need to change project automation in ``cumulusci.yml``. Included Libraries ================== CumulusCI comes bundled with additional third-party keyword libraries, in addition to the libraries that come with Robot Framework itself: * `SeleniumLibrary `_ for browser testing * `RequestsLibrary `_ for testing REST APIs SeleniumLibrary is automatically imported when you import ``Salesforce.robot``. To use ``RequestsLibrary`` you need to explicitly import it in the settings section of your Robot test. Example Robot Test ================== The following test file placed under ``robot/ExampleProject/tests/create_contact.robot`` in your project's repository automates the testing of creating a Contact through the Salesforce UI in a browser and via the API. As an added convenience, it automatically deletes the created Contacts in the Suite Teardown step: .. code-block:: robotframework *** Settings *** Resource cumulusci/robotframework/Salesforce.robot Library cumulusci.robotframework.PageObjects Suite Setup Open Test Browser Suite Teardown Delete Records and Close Browser *** Test Cases *** Via API ${first_name} = Get fake data first_name ${last_name} = Get fake data last_name ${contact_id} = Salesforce Insert Contact ... FirstName=${first_name} ... LastName=${last_name} &{contact} = Salesforce Get Contact ${contact_id} Validate Contact ${contact_id} ${first_name} ${last_name} Via UI ${first_name} = Get fake data first_name ${last_name} = Get fake data last_name Go to page Home Contact Click Object Button New Wait for modal New Contact Populate Form ... First Name=${first_name} ... Last Name=${last_name} Click Modal Button Save Wait Until Modal Is Closed ${contact_id} = Get Current Record Id Store Session Record Contact ${contact_id} Validate Contact ${contact_id} ${first_name} ${last_name} *** Keywords *** Validate Contact [Arguments] ${contact_id} ${first_name} ${last_name} [Documentation] ... Given a contact id, validate that the contact has the ... expected first and last name both through the detail page in ... the UI and via the API. # Validate via UI Go to page Detail Contact ${contact_id} Page Should Contain ${first_name} ${last_name} # Validate via API &{contact} = Salesforce Get Contact ${contact_id} Should Be Equal ${first_name} ${contact}[FirstName] Should Be Equal ${last_name} ${contact}[LastName] Settings -------- The Settings section of the ``.robot`` file sets up the entire test suite. By including the Resource ``cumulusci/robotframework/Salesforce.robot``, which comes with CumulusCI, we inherit a lot of useful configuration and keywords for Salesforce testing automatically. The Suite Setup and Suite Teardown are run at the start and end of the entire test suite. In the example test, we're using the ``Open Test Browser`` keyword from the ``Salesforce.robot`` file to open a test browser. We're also using the ``Delete Records and Close Browser`` keyword from ``Salesforce.robot`` to automatically delete all records created in the org during the session and close the test browser. Test Cases ---------- The two test cases test the same operation done through two different paths: the Salesforce REST API and the Salesforce UI in a browser. Via API ^^^^^^^ This test case uses the ``Get fake data`` keyword to generate a first and last name. It then uses the ``Salesforce Insert`` keyword from the Salesforce Library (included via ``Salesforce.robot``) to insert a Contact using the same technique for generating test data. Next, it uses ``Salesforce Get`` to retrieve the Contact's information as a dictionary. Finally, the test calls the ``Validate Contact`` keyword explained in the Keywords section below. Via UI ^^^^^^ This test case also uses ``Get fake data`` for the first and last name, but instead uses the test browser to create a Contact via the Salesforce UI. Using keywords from the Salesforce Library, it navigates to the Contact home page and clicks the ``New`` button to open a modal form. It then uses ``Populate Form`` to fill in the First Name and Last Name fields (selected by field label) and uses ``Click Modal Button`` to click the ``Save`` button and ``Wait Until Modal Is Closed`` to wait for the modal to close. At this point, we should be on the record view for the new Contact. We use the ``Get Current Record Id`` keyword to parse the Contact's ID from the URL in the browser and the ``Store Session Record`` keyword to register the Contact in the session records list. The session records list stores the type and Id of all records created in the session, which is used by the ``Delete Records and Close Browser`` keyword on Suite Teardown to delete all the records created during the test. In the ``Via API`` test, we didn't have to register the record since the ``Salesforce Insert`` keyword does that for us automatically. In the ``Via UI`` test, we created the Contact in the browser and thus need to store its ID manually for it to be deleted. Keywords -------- The ``Keywords`` section allows you to define keywords useful in the context of the current test suite. This allows you to encapsulate logic you want to reuse in multiple tests. In this case, we've defined the ``Validate Contact`` keyword which accepts the Contact id, first, and last names as argument and validates the Contact via the UI in a browser and via the API via ``Salesforce Get``. By abstracting out this keyword, we avoid duplication of logic in the test file and ensure that we're validating the same thing in both test scenarios. Running the Test Suite ---------------------- This simple test file can be run via the ``robot`` task in CumulusCI: .. code-block:: console $ cci task run robot -o suites robot/MyProject/tests/create_contact.robot -o vars BROWSER:firefox 2019-04-26 09:47:24: Getting scratch org info from Salesforce DX 2019-04-26 09:47:28: Beginning task: Robot 2019-04-26 09:47:28: As user: test-leiuvggcviyi@example.com 2019-04-26 09:47:28: In org: 00DS0000003ORti 2019-04-26 09:47:28: ============================================================================== Create Contact ============================================================================== Via API | PASS | [ WARN ] Retrying call to method _wait_until_modal_is_closed ------------------------------------------------------------------------------ Via UI | PASS | ------------------------------------------------------------------------------ Create Contact | PASS | 2 critical tests, 2 passed, 0 failed 2 tests total, 2 passed, 0 failed ============================================================================== Output: /Users/boakley/dev/MyProject/robot/MyProject/results/output.xml Log: /Users/boakley/dev/MyProject/robot/MyProject/results/log.html Report: /Users/boakley/dev/MyProject/robot/MyProject/results/report.html .. note:: In the example output, the WARN line shows functionality from the Salesforce Library which helps handle retry scenarios common to testing against Salesforce's Lightning UI. In this case, it automatically retried the wait for the modal window to close after creating a contact in a browser. If you put all of your tests inside that ``robot//tests`` folder you don't have to use the ``suite`` option. By default the ``robot`` task will run all tests in the folder and all subfolders. For example, to run all tests and use the default browser you just have to issue the command ``cci task run robot``. ``Salesforce.robot`` ==================== Keywords can be defined in a test suite file, but they can also be defined in libraries and resource files. Libraries are written in Python, and resource files are written in the Robot syntax. Resource files are almost identical to a test file, except that they have no tests and can be imported into other test files. In addition to containing keywords, resource files can also define variables and can import other libraries. The file ``cumulusci/robotframework/Salesforce.robot`` was designed to be the way to import all of the keywords and variables provided by CumulusCI. It should be the first item imported in a test file. It will import the :ref:`salesforce-library-overview` and :ref:`cumulusci-library-overview`, as well as the most commonly used robot libraries (`Collections `_, `OperatingSystem `_, `String `_, and `XML `_) Variables defined in resource files are accessible to all tests in a suite which imports the resource file. They can be set in your cumulusci.yml file, or specified with the ``vars`` option to the robot task. When doing so, the variables need to be referenced without the dollar sign and curly braces. Variable names are case-insensitive. For example, here is how to set the browser to Firefox and the default timeout to 20 seconds in a ``cumulusci.yml`` file: .. code-block:: yaml tasks: robot: options: vars: - BROWSER:firefox - TIMEOUT:20 seconds The same variables can be set from the command line to override the config file for a single test run. This example shows that you can use the lowercase name for convenience: .. code-block:: console $ cci task run robot -o vars browser:firefox,timeout:20 Supported Variables ------------------- The following variables defined in ``Salesforce.robot`` are all used by the ``Open Test Browser`` keyword: .. list-table:: :widths: 1 3 * - ``${BROWSER}`` - Defines the browser to be used for testing. Supported values are ``chrome``, ``firefox``,`` headlesschrome``, and ``headlessfirefox``. Default: ``chrome`` * - ``${DEFAULT_BROWSER_SIZE}`` - This sets the preferred size of the browser. It is specified in the form of widthxheight, and the values are passed to the `Set window size `_ keyword. Default: ``1280x1024`` * - ``${IMPLICIT_WAIT}`` - This is automatically passed to the `Set Selenium Implicit Wait `_ keyword. Default: ``7 seconds`` * - ``${SELENIUM_SPEED}`` - This defines a delay added after every Selenium command. It is automatically passed to the `Set Selenium Speed `_ keyword. Default: ``0 seconds`` * - ``${TIMEOUT}`` - This sets the default amount of time Selenium commands will wait before timing out. It is automatically passed to the `Set Selenium Timeout `_ keyword. Default: ``30 seconds`` Full Documentation ------------------ To see the full list of keywords and their descriptions for this library, see the section titled `Salesforce.robot `_ in the keyword documentation. .. _cumulusci-library-overview: CumulusCI Library ================= The CumulusCI Library for Robot Framework provides access to CumulusCI's functionality from inside a Robot test. It is mostly used to get credentials to a Salesforce org and to run more complex automation to set up the test environment in the org. Logging Into An Org ------------------- The ``Login Url``* keyword returns a url with an updated OAuth access token to automatically log into the CumulusCI org from CumulusCI's project keychain. Run Task -------- The ``Run Task`` keyword is used to run named CumulusCI tasks configured for the project. These can be any of CumulusCI's built in tasks as well as project specific custom tasks from the project's cumulusci.yml file. ``Run Task`` accepts a single argument, the task name. It optionally accepts task options in the format ``option_name=value``. Run Task Class -------------- The ``Run Task Class`` keyword is for use cases where you want to use one of CumulusCI's Python task classes to automate part of a test scenario but don't want to have to map a custom named task at the project level. ``Run Task Class`` accepts a single argument, the ``class_path``, as it would be entered into ``cumulusci.yml``, such as ``cumulusci.tasks.salesforce.Deploy``. Like ``Run Task``, you can also optionally pass task options in the format ``option_name=value``. Set Test Elapsed Time --------------------- This ``Set Test Elapsed Time`` keyword captures a computed rather than measured elapsed time for performance-tests. For example, if you were performance testing a Salesforce batch process, you might want to store the Salesforce-measured elapsed time of the batch process instead of the time measured in the CCI client process. The keyword takes a single optional argument which is either a number of seconds or a Robot time string (https://robotframework.org/robotframework/latest/libraries/DateTime.html#Time%20formats). Using this keyword will automatically add the tag cci_metric_elapsed_time to the test case. Performance test times are output in the CCI logs and are captured in MetaCI instead of the "total elapsed time" measured by Robot Framework. Start and End Perf Time ----------------------- As a convenience, there are keywords to handle the common case where you want to start a timer and then store the result with ``Set Test Elapsed Time``. These are ``Start Performance Timer`` and ``Stop Performance Timer``. Set Test Metric --------------- This keyword captures any metric for performance monitoring. For example: number of queries, rows processed, CPU usage, etc. Elapsed Time For Last Record ---------------------------- The ``Elapsed Time For Last Record`` queries Salesforce for a value that is Salesforce's recorded log of a job. For example, to query an Apex bulk job: .. code-block:: robot ${time_in_seconds} = Elapsed Time For Last Record ... obj_name=AsyncApexJob ... where=ApexClass.Name='BlahBlah' ... start_field=CreatedDate ... end_field=CompletedDate ... order_by=CompletedDate Full Documentation ------------------ To see the full list of keywords and their descriptions for this library, see the section titled `CumulusCI `_ in the keyword documentation. .. _salesforce-library-overview: Salesforce Library ================== The Salesforce Library provides a set of useful keywords for interacting with Salesforce's Lightning UI and Salesforce's APIs to test Salesforce applications. In addition to keywords, the library defines some custom locator strategies to aid in locating elements on a page. UI Keywords ----------- The goal of the UI keywords in the Salesforce Library is to abstract out common interactions with Salesforce from interactions with your application's UI. The Salesforce Library itself has an extensive suite of Robot tests which are regularly run to alert us to any changes in the base Salesforce UI. By centralizing these interactions and regularly testing them, the Salesforce Library provides a more stable framework on which to build your product tests. There are too many keywords relating to UI interactions to cover here. Please reference the full Salesforce Library documentation below. Waiting for Lightning UI ^^^^^^^^^^^^^^^^^^^^^^^^ A common challenge when writing end-to-end UI tests is the need to wait for asynchronous actions to complete before proceeding to run the next interaction. The Salesforce Library is aware of the Lightning UI and can handle this waiting automatically. After each click, it will wait for any pending requests to the server to complete. (Manually waiting using a "sleep" or waiting for a particular element to appear may still be necessary after other kinds of interactions and when interacting with pages that don't use the Lightning UI.) API Keywords ------------ In addition to browser interactions, the Salesforce Library also provides the following keywords for interacting with the Salesforce REST API: * ``Salesforce Collection Insert``: used for bulk creation of objects based on a template. * ``Salesforce Collection Update``: used for the bulk updating of objects. * ``Salesforce Delete``: Deletes a record using its type and ID. * ``Salesforce Get``: Gets a dictionary of a record from its ID. * ``Salesforce Insert``: Inserts a record using its type and field values. Returns the ID. * ``Salesforce Query``: Runs a simple query using the object type and field=value syntax. Returns a list of matching record dictionaries. * ``Salesforce Update``: Updates a record using its type, ID, and field=value syntax. * ``SOQL Query``: Runs a SOQL query and returns a REST API result dictionary. Locator Strategies ------------------ SeleniumLibrary provides many locator strategies for finding elements on a page. For example, you can specify an element via an xpath, an id, a css selector, or several others. These are documented in the SeleniumLibrary documentation under a section titled `Locating elements `_. In addition to the predefined locator strategies, the Salesforce library defines the following locator strategies, all of which use keywords in the Salesforce library to find web elements. For detailed explanations of the locator strategies, see the documentation for each keyword. .. list-table:: :widths: 1 3 * - ``label`` - This uses the Salesforce library keyword ``Locaate Element by Label`` to find web elements. It is most useful to find form fields based on lightning web components and which have a ``label`` associated with the component. For example, ``label:First Name`` might return a ```` component that wraps a block of code which contains a ``