July saw the conclusion of the Advanced In-The-Wild Malware Testing series, in which we use malware found ‘in the wild’ on the web to perform security tests. Using a honeypot network and access to public sources with links to malware, we qualify tested, working samples for testing. We reject samples that are unsuitable for testing for a variety of reasons, such as lack of correct operation in Windows, lack of malicious activity, lack of communication with the hacker’s server. Well, in July 2024, out of almost a thousand samples, 275 were qualified for testing.
Our tests comply with the guidelines of the Anti-Malware Testing Standards Organization. Details about the test are available at this website as well as in our methodology.
Summary of the July 2024 tests
We tested 14 solutions and found that only ESET Smart Security did not receive the maximum score, as in one case it failed to correctly detect and block a threat on default settings with LiveGrid protection enabled (configuration details below).
We would like to point out that when analysing the study, the Remediation Time parameter should be taken into account: this is the average time it takes for a given threat to be completely neutralized from the time the file enters the system until it is blocked by the antivirus product (neutralization of the threat after launch). The more threats stopped in the pre_execution phase, the lower the Remediation Time parameter will be in terms of average time in seconds.
Each launch of the malware requires its analysis, which increases the average remediation time. Success is guaranteed by a 100% response to the threats used in the test, which is the most important factor for the user.
Detailed results can be found on the recent results page dedicated to Advanced In-The-Wild Malware analysis.
The fastest vendors to neutralise and remove threats correctly were:
- Bitdefender Total Security (took 0.01 seconds on average on the set of samples used)
- Kaspersky Plus (0.715s)
- F-Secure Total (0.799s)
Settings of the tested solutions
Although we avoid using PUP.PUA samples in the test, it is always a good idea to have such protection enabled, and we use such settings.
Predefined settings are good, but not always the best. Therefore, for the sake of full transparency, we list the ones we have implemented for better protection, or if required by the vendor.
In addition to the changes listed below, we always configure the solution to use a special extension for the Firefox browser we are testing (if the extension is available).
Second, we configure the software to automatically block, remove and remediate incidents.
The configuration of the software is as follows:
- Avast Free Antivirus: default settings + automatic PUP repair + browser protection.
- Bitdefender Total Security: default settings + crypto miner detection + browser protection.
- Comodo Internet Security 2025: default settings + do not show an alert for sandbox (block).
- Emsisoft Enterprise Security: default settings + automatic PUP repair + EDR + Rollback + browser protection.
- Eset Smart Security Premium: default settings + browser protection +
- LiveGrid enabled.
- PUA detection enabled.
- LiveGuard set to âend process and cleanâ.
- Protection and detection at the âbalancedâ level.
- F-Secure Total: default settings + browser protection.
- Kaspersky Plus: default settings.
- Malwarebytes Premium: default settings + browser protection.
- McAfee Total Protection: default settings + browser protection.
- ThreatDown Endpoint Protection: default settings + EDR + browser protection.
- Quick Heal Total Protection: default settings.
- Panda Dome: default settings:
- Webroot Antivirus: default settings + browser protection.
- Xcitium ZeroThreat Advanced: predefined policy -> Windows Secure Profile 8.1 + EDR.
Update 22 August 2024 – what has changed?
On 19 August 2024, two days after the missed deadline for providing feedback on the technical data from the Advanced In-The-Wild Malware test, we received evidence from an ESET representative that a malware sample had been detected which, according to our algorithms, was initially missed by ESET Smart Security software on 14 July 2024 when the sample was tested at that time.
As a tester at AMTSO (Anti-Malware Testing Standard Organisation), we would like to bring this to the public’s attention. After reviewing the evidence, we believe that ESET’s request is legitimate. In the interest of full transparency, we also include the following technical data and logs:
SHA256 sample checksum:
5fb58f1f2e8e61c2a67fb5a8a14b9a5e676fc52a8e7e864a9e1bdb17d1dcf1b2
Exact start date of sample test:
2024-07-14 05:10:57.463
Detailed test logs in ZIP format (password: infected):
[ download ]
After downloading and extracting the archive, the following logs will be available, among others:
- a. output.txt file as a real-time console record of the actions performed by the AVLab testing system
- b. malware sample with the original link where it was downloaded from
- c. XML Sysmon report
- d. /ProgramData/ESET/ESET Security/Logs/warnlog.dat <- the following file contains information that the sample was transferred to the Eset cloud for analysis during the test. This is sufficient evidence of sample detection:
The Eset software stores logs in encrypted form. In order to view them, you will need to download the Eset Log Collector software and preview the specified file.
We have complied with the ESET’s request to include this evidence for a sample that was initially incorrectly marked as Fail. As a result, the score for the Eset software has been corrected to 100% blocking of all malware samples used in the Advanced In-The-Wild Malware Test in the July 2024 edition.
