The CheckLab organization was founded in July 2019 by the AVLab.pl company that has been operating since 2012 in the industry of cybersecurity. The primary objective of the CheckLab organization is to test the usefulness of workstations security, and issuing certificates confirming the protection effectiveness against malicious software, and also provide results to public information while ensuring the maximum transparency of the tests. In the tests, the CheckLab employees use malicious software, tools, and techniques of bypassing security that are used in real cyberattacks.
Introduction to the „Advanced In The Wild Malware Test”
The name of the „Advanced In The Wild Malware Test” perfectly reflects its character. The source of malicious software are honeypots located on all continents of the world. We collect malware, among other, for the Windows system. Samples captured in attacks are checked on the basis of over 100 patterns before they are qualified for testing. These patterns allow us to determine whether a potentially dangerous file is actually a threat to the Windows 10 Pro operating system.
In July 2019, in the second edition of security tests of the „Advanced In The Wild Malware Test” we verified the effectiveness of detecting and blocking malicious software of 8 solutions for protecting computers.
The tests lasted continuously the whole September 2019. The list of tested solutions is as follows:
- AVAST Free Antivirus
- AVIRA Antivirus Pro
- COMODO Internet Security
- COMODO One
- KASPERSKY Endpoint Security Cloud Plus
- SECUREAPLUS Premium
- WEBROOT SecureAnywhere Antivirus
- ZONE ALARM Extreme Security (Check Point)
The results of September 2019
CheckLab as the first organization in the word shows such detailed information from tests to all interested people. We share checksums of malicious software by dividing them into protection technologies that have contributed to detect and stop a threat. According to experts, this type of innovative approach of comparing security will contribute to better understanding of differences between available products for consumers and enterprises.
The chart of level blocking are available at https://avlab.org/recent-results:
In the second edition of the test, we have granted the BEST+++ certificate to:
- Avast Free Antivirus
- Comodo Internet Security
- Comodo ONE
- Kaspersky Endpoint Security Cloud Plus
- SecureAPlus Premium
- Webroot Antivirus
- ZoneAlarm Extreme Security
We have granted the certificate BEST++ to:
- Avira Antivirus Pro
We haven’t granted the certificate GOOD+.
Levels of blocking malicious software samples
The CheckLab employees are probably pioneers in this regard — they show more detailed diagnostic data than any other testing institution, including the largest such as AV-Comparatives and AV-Test). Blocking of each malware sample by tested protection solution has been divided into a few levels:
- Level 1 (P1): The browser level, i.e. a virus has been stopped before or after it has been downloaded onto a hard drive.
- Level 2 (P2): The system level, i.e. a virus has been downloaded, but it has not been allowed to run.
- Level 3 (P3): The analysis level, i.e. a virus has been run and blocked by a tested product.
- Failure (N): The failure, i.e. a virus has not been blocked and it has infected a system.
The result of blocking each sample are available at https://checklab.pl/en/recent-results in the table:
The products and Windows 10 settings: daily test cycle
Tests are carried out in Windows 10 Pro x64. The user account control (UAC) is disabled because the purpose of the tests is to check the protection effectiveness of a product against malware and not a reaction of the testing system to Windows messages.
Additionally, the Windows 10 system contains installed the following software: office suite, document browser, email client, and other tools and files that give the impression of a normal working environment.
Automatic updates of the Windows 10 system are disabled in a given month of the tests. Due to the complicated process and the possibility of a malfunction, Windows 10 is updated every few weeks under close supervision.
Security products are updated one time within a day. Before tests are run, virus databases and protection product files are updated. This means that the latest versions of protection products are tested every day.
Malicious software
We have used 633 malicious software samples for the test, consisting of, among others, banking trojans, ransomware, backdoors, downloaders, and macro viruses. In the contrast to the well-known institutions that verify the security usefulness, the CheckLab tests are much more transparent because the organization share the full list of malware samples.
During testing, all solutions have access to the Internet. The experts of CheckLab use real working environments in a graphic mode that is why the results of individual samples may differ from those presented by the VirusTotal service. The CheckLab organization points that out because inquisitive users may compare our tests with the scanning results of VirusTotal. It turns out that differences between real products installed on Windows 10 and scanning engines on VirusTotal are significant.
