Development of autotests for CRM-system

RetailCRM project is rapidly growing since 2013. When the code amount is hundreds of thousands lines, inevitably there is a question of keeping system work stability when intensive developing of new product features.

For this purpose the full cycle of product testing was implemented, which allows to save the stability of retailCRM working version and to update system twice a week imperceptibly and without serious consequences for users and with minimum developers time consumption.

TASK
Saving of retailCRM project stability when intensive developing.

Automatic checking of new retailCRM features at the moment of their developing and implementation to working version of system.

WHAT WE DID
We have implemented a complete testing cycle of retailCRM

About the project
RetailCRM project started in 2013. Then it was internal startup with the limited functionality, not requiring the testing.

One year later the project became larger and we have started to think about the project testing. In parallel there was a task to update versions of Symfony framework, on which the product is developed. That time the system were working on Symfony 2.1, when Symfony 2.5 was already launched.

It was needed to update Symfony for keeping up to date and not to allow the project code to become outdated. Updating was followed by serious risks, since versions have differences and there was a possibility to get a lot of errors, so it was the cause for writing the first set of tests.

First, the basic tests were written, which just passed by all system pages, sent forms, filters, switched the sorting and checked the absence of errors 500, it allowed to find out the most obvious breakdowns after updating.

Next series of tests were already checking the work correctness of crucial pages and forms. The logic, which independently passes by all sortings of columns and lists, all filters, was added, it fills in the form with different data and checks their work correctness.

Also there were added the tests for business logic, for statuses transition, checking of rights for access to sections. All this functionality was written for one month. In July 2015 it was decided that all written tests (around 300 at that moment) are enough for system updating.

After Symfony updating we started to write the tests mandatorily for all tasks, in passing creating tests for old tasks.

The more tests were created, the longer they were executed. When it was only first series of tests, they were executed just for several minutes. They were executed locally.

When the tests` execution began to take tens of minutes, waiting for execution were very inconvenient for developer. It was decided to use PHPCI, the system, which allows to execute the tests automatically. PHPCI integration with git and Redmine, which allows to work autonomously, was made.

AUTOTESTS — are the scenarios, which check the work correctness of small blockes of code or imitate the user interaction with service. Autotests allow to avoid manual testing and to exclude the code breakdown when adding new functionality to the system.
Meaning of three-sided integration of PHPCI with git and Redmine
1. There is a task, executing by developer in new branch at git.

2. The same number, as a task has, is assigned to this branch.

3. Developer realizes the task, adds tests on it, then executes commit and push to the central repository on server.

4. PHPCI monitors automatically the appearance of new branches and new code on the server.

5. PHPCI catches up the new branch, takes project from there, opens it and starts to execute tests.
6. If all tests are passed, then PHPCI notifies about successful tests passing: it changes task status to "Feedback" and 80% of readiness automatically.

7. If the result is negative (at least one test was not passed), status changes automatically that tests were not passed. There is a link in message from PHPCI.

8. PHPCI executes tests for all PHP versions, the project must work at, and notes the result of each execution in the task.
Task realization cycle from definition till deployment to working CRM-system with using of autotests
Autotests applying to another systems
At the moment autotests are mostly used in retailCRM main service. But there are also infrastructure services, which solve related tasks in the project, like:

  • Billing;
  • Deployer;
  • Geohelper;
  • Daemon Collector etc.
There are certain tests for each project.

Tests were executed for all major versions of language supported by project. At the moment the minimum retailCRM requirements are ≥ PHP5.6, and each task is executed for PHP5.6 and PHP7.
Data, checked when autotesting
1.
Upload of unit-tests for API
2.
Functional tests for business logic
Functional tests are much heavier than unit-tests. They require the reloading of test fixtures to database and emulate full request/response-logic. For example, CI has begun from medium virtual machine on 4 kernels and tests execution has been taking 10-20 minutes, and for two years the execution time increased to 3-5 hours. After optimization series and move to dedicated server the time was reduced to 45 minutes.
Achieved results
1. Solving of localization errors problem when updating of frameworks and libraries, used in project. Updating of Symfony version carries out in regular mode with code customization in places of test failures.

2. Reducing of code checking time for technical leader.

3. Fast and stable implementation of new features.

I think that tests` main goal is not to check work of new functionality at the moment of its development (though this too), but to provide working capacity of existing functionality when developing the new one. When the code amount is hundreds of thousands lines, only tests can help to keep the project work stability when its active developing.

As of August, 2016 more than 2000 tests are already developed. There are about 40 thousand different checks for this or that logic of retailCRM service work in these tests. Without implementation of autotests it would be almost impossible.
Ilyas Salikhov
CTO, RETAILCRM
+7 495 268-02-56
We are in web-development business since 2007. We rank among Top 100 RuNet web-developers and Top 5 e-shop developers.
Khlebozavodskiy proezd 7, Unit 9, Moscow, 115230
Got interesting projects? Contact us!