Neotys develops a software testing tool designed to improve the quality and performance of information systems, its primary product^[buzzword], NeoLoad, is the most automated performance testing platform for enterprise organizations continuously testing from APIs to applications. Neotys is a privately owned company.^[1]

Neotys develops a software testing tool designed to improve the quality and performance of information systems, its primary product buzzword, NeoLoad, is the most automated performance testing platform for enterprise organizations continuously testing from APIs to applications. Neotys is a privately owned company. NeoLoad can collect end-user experience and sends to NeoLoad the metrics calculated on the client (browser or mobile devices). We don’t have any dedicated integration for Karma, but we can definitely support it and reach the same level of benefit we have with Selenium. NeoLoad is the fastest and the most automated load testing tool on the market that helps you eliminate bottlenecks in all your web and mobile applications.

History[edit]

Neotys was founded in 2005 by a group of software developers and IT project managers who recognized demand for a new software solution^[buzzword] to test the performance of web applications before launch. Since then, Neotys has delivered software solutions^[buzzword] to more than 2000 clients in 70 countries, with exports accounting for more than 75% of the company's annual turnover.

References[edit]

^{[2][3][4][5][6][7][8][9]}

1. ^'The Load Testing Platform Accelerating DevOps | NeoLoad by Neotys'. Retrieved 2018-12-17.
2. ^Neotys Receives the 2019 Technology Alliance Award from Dynatrace [1], Neotys, 2019
3. ^2019 Gartner Peer Insights “Voice of the Customer” Report for Software Test Automation [2], Gartner, 2019
4. ^Neotys Recognized by CIOReview as Promising Software Testing Solution [3], Neotys, 2019
5. ^NeoLoad is an SAP Certified Technology [4], Neotys, 2019
6. ^2018 DevOps Showcase [5], SD Times, 2018
7. ^The 2018SD Times 100: 'Best in Show' in Software Development [6], SD Times, 2018
8. ^Neotys Receives the 2018 Solution Innovation Award from Dynatrace [7], Neotys, 2018
9. ^Neotys Receives the 2018 Top Partner Collaborator Award from QASymphony [8], Neotys, 2018

This command-line interface helps you launch and observe performance tests on the Neotys Platform. Since NeoLoad is very flexible to many deployment models (SaaS, self-hosted, cloud or local containers, etc.), configuration and test execution parameters depend on your licensing and infrastructure provisioning options. Please read the following instructions carefully.

• Additional Options

Prerequisites

The examples below assume that you have Python3 and Git command line tools installed.

For Windows 10 users, see:

• Installing Python in WindowsIn short:
  • Just install via Python.org Downloads, then
  • Open a command prompt and install pip:
• Install Docker with Chocolatey
  • Open a command prompt and install Docker Desktop for Windows 10

For Mac OS X:

For Docker builds:

Installation

To install, simply run the following command. As of Jan 2020, Python 2 will be permanently deprecated, therefore this utility is written for Python 3.

Interactive CLI Help

For usage and CLI argument examples, simply run:

Configuration

Before running a load test, you will need to initialize a profile to contain connection information to NeoLoad Web. Subsequent commands will use this profile unless switched to another profile, as there can be multiple. This is so that you only need to specify these details once only, then you can focus on validating and running load tests.

Your profile details depend on your license model and how you want to obtain load testing resources (controller + load generators). See below:

Create a SaaS-Based License Profile

If you have an enterprise license activated in NeoLoad Web SaaS, you do not need to specify license details in your profile. When a controller begins to run a test, it will reach out to NeoLoad Web SaaS to lease an appropriate license.

Create a Self-Hosted Enterprise License Profile

If you do not have a SaaS-based license, you will need to specify additional licensing server url and credentials that refer to your own NeoLoad Team License Server.

Viewing saved profiles

You can always list profiles that have saved using the following command:

You can also view the raw/complete stored JSON representation of saved profile details using the following command:

Future Plans to Execute Local Tests on a Free Trial License

If you do not already have an enterprise license, such as if you only have NeoLoad installed on your local workstation (from a Free Trial download or professional license), NeoLoad CLI will eventually support using your installation as an attached controller and load generator.

Additionally, if you do not have access to a dynamic infrastructure provider (i.e. OpenShift, etc.) and if you do not have Docker installed where the NeoLoad CLI is executed, your local workstation can serve as load testing resources similar to dynamic and containerized resource use cases.

This is planned to be delivered in Dec 2019.

Running Load Tests

Once a profile is established, NeoLoad CLI makes it very easy to execute load tests. All you need is to provide an existing test suite or set of as-code file(s), then specify a scenario.

NeoLoad CLI defaults to using the NeoLoad Web APIs for Runtime operations, which means that your project assets will be zipped up together and uploaded to the NeoLoad Web deployment you specified in your profile (SaaS or self-hosted).

Once a test is initialized, if you are running in interactive console mode, the NeoLoad CLI will automatically open the system default browser to your live test results.

Obtain Basic Examples

Some basic examples are in our Git repository for this utility, under the directory ./tests/. To get them, simply clone the repo:

Then, underneath this directory, you can run them simply by typing in:

Neoload 7.8.0

Additionally, you can specify multiple files, such as additional SLA, variables, or servers overriding files. This works for both [.nlp] and [.yaml] files.

Non-blocking Execution Workflow

You may want to manage individual steps of attaching resources, running, waiting, and detatching resources in a parallel pipeline. This is particularly useful in combination with other parallel steps that dynamically analyze SLA data and fast-fail the test (using API commands) using a known custom real-time analysis process. Worlds together worlds apart 3rd edition volume 2.

The general process can be seen in the NeoLoad CLI E2E PyTest suite, but also abstracted below:

• Initialize a profile

• (Optional for static zone) Attach dynamic resources

  (Note: this assumes that the host you run the above command is a Docker host or is connected to one, DinD)

• Kick off a test via NeoLoad Web Runtime

• Grab the ID of the test just executed

• (Start some other parallel operations, such as monitoring for early advanced SLA failures)

• Wait (blocking) for test to complete

• (Always) Detatch Optional Docker resources

Additional Options

There are many other arguments for test summarization, modification, and exporting results.

Exporting SLA Results to JUnit

Verbose and Debug Mode

Verbose mode changes logging level to INFO. When executing from an interactive console, it also opens a browser window to the test logs immediately after they're available (useful for monitoring the infrastructure initialization process).

Debug mode changes logging level to DEBUG. This is an extreme amount of internal information which infers verbose/INFO mode and provides manual pausing on critical events (such as after Docker attach but before test execution).

Quiet Mode

Adding the --quiet flag generally produces only the relevant output (i.e. JSON, test id, etc.) and no other informational messages. This is useful when piping the structured data output into another process, such as obtaining the most recent test id from stdout file (--quiet is inferred by some other combinations of flags such as --infile AND --query testid).

Test Summary

Specifying both a --testid value and the --summary flag produces a JSON result set that shows both the high level test status metadata and overall statistics.

Produces:

TODO: This summary will also include SLA summaries in Jan 2020

Test Modifications

Metadata on a test can be modified after the test is complete, such as name, description, and status. With the required --testid argument as well, these modification arguments can be used individually or in combination with each other.

Additionally, plus and minus (+/-) operators can be used to append or remove the text specified from the existing name or description data. This is useful in combination with hashtag keywords to flag a test as baseline, candidate, or with an issue tracking id.

Test Result listings and queries

TODO: will be coming in Jan 2020

Contributing

Feel free to fork this repo, make changes, test locally, and create a pull request. As part of your testing, you should run the built-in test suite with the following command:

Neoload Vs Jmeter

*NOTE: omitting the --skipslow and --skipslas arguments also runs Docker-related attaching tests, which you will need to set environment variables up for in order to successfully run the test suite. An example of these variables can be found in example.bash_profile and can be addapted for Windows execution as well.