Anti-Phishing Test Methodology

Operating system / Browser

Phishing tests may be carried out on Microsoft Windows, macOS, Android, or iOS platforms. Details of the exact operating system version, architecture, browser(s), and test environment used will be provided in the individual test reports.

Please note that phishing tests can be carried out on the anti-phishing features built into individual browsers, without an additional security product, or on the anti-phishing measures provided by security products. Hence the use of the term “browser/security product” throughout this document.

The browser(s) and operating system(s) used in each test will be specified in the corresponding test report.

Aim of the test

The test is intended to demonstrate how effective the participating browser/security products are at recognising and blocking phishing websites, and thus protecting the user from being defrauded by these sites.

Target Audience

Any computer user who does not feel completely confident of their own ability to recognise and avoid phishing attacks will benefit from using a security product/browser with effective phishing protection. Any computer enthusiast or professional who provides technical support for family, friends, colleagues or clients will also be concerned with installing or recommending products that provide phishing protection for their supported users.

Definition of the threat

A phishing site is a website that attempts to impersonate a legitimate person, organisation, brand, service, platform, or website, and aims to deceive users into disclosing credentials, personal information, financial information, authentication codes, payment details, identity documents, or other sensitive data, or into performing actions that may result in fraud, identity theft, account compromise, financial loss, or other criminal activity.

One very common type of phishing attack involves sending out spam mails purporting to be from a bank, with a message to recipients that they need to log on to their Internet banking account for one reason or another. A hyperlink is provided in the mail, supposedly giving the victims easy access to their online accounts. In reality, the link leads to a fake copy of the bank’s login page. This will capture the user’s login credentials, which can then be used by the perpetrators of the scam.

However, modern phishing attacks are not limited to credential theft. Phishing websites may also attempt to collect personal information, payment card details, banking information, one-time passwords (OTPs), recovery codes, government-issued identity information, cryptocurrency wallet details, or other sensitive information. Such sites are considered phishing within the scope of this test, even when no account credentials are requested.

There are numerous types of online fraud that are not considered phishing and are consequently not included in this test. Examples include fraudulent investment schemes, fake online stores, technical-support scams, advance-fee fraud, romance scams, or misleading websites that do not impersonate an existing trusted entity and whose primary purpose is not the collection of sensitive information through deception.

Many web-based malware attacks use legitimate web servers to host malware executables. Equally, it is possible for phishing attacks to host their webpages on the servers of reputable organisations that have been compromised. A phishing page should be recognised regardless of where it is hosted, though if a legitimate top-level domain is blocked, this would be regarded as a false positive. For example, if a phishing page is hosted under the URL www.lycos.com/user2035/personal/index.htm, this particular URL should be blocked, but blocking lycos.com (a legitimate domain) would be a false positive.

Scope of the test

The test is optional; vendors who have joined the main-test series can decide whether or not to participate. Our phishing-protection test evaluates the ability of a browser or security product to identify and warn users about phishing websites. The focus is on the detection and blocking of phishing URLs and webpages that attempt to obtain credentials, personal information, financial information, authentication codes, identity-related information, or other sensitive data through impersonation or deception.

The test evaluates protection at the URL and webpage level. Products are expected to detect or block phishing pages before any sensitive information is entered or submitted. Protection mechanisms that rely solely on analysing data entered by the user, monitoring outbound submissions, or intervening only after information has been entered are outside the scope of this test.

As noted above, phishing attacks commonly use links in spam emails, text messages, social-media messages, advertisements, QR codes, or compromised websites to persuade users to visit phishing pages. The delivery vector leading to the phishing URL is not considered in this test. The test focuses exclusively on the browser’s or security product’s ability to identify the phishing page itself.

Test Setup

The test is carried out on identical test systems using a standardised configuration.

The operating system(s) and browser(s) to be used in a test will be announced to participating vendors before testing begins. Depending on the scope of the test and product availability, testing may be conducted on Microsoft Windows, macOS, Android, and/or iOS platforms.

The browser used will be a popular mainstream browser that is supported by all participating products. For products that integrate directly into the operating system or browser, the vendor-recommended configuration will be used.

Identical operating system and browser configurations are installed on all test systems. Any built-in phishing protection mechanisms that are not part of the product under test are disabled where technically feasible and appropriate. For security-product tests, browser-integrated phishing protection features are disabled whenever possible to ensure that the measured protection is provided by the product under test. In browser-only tests, the browser’s native phishing protection remains enabled.

Each product is installed, updated to the latest available version, and configured using its default settings unless otherwise specified in the test report.

Settings

All settings are left at their default values. The products have unrestricted cloud access throughout the test. Before the test proper is run, all products will be tested to ensure that they are correctly configured and functioning properly.

Sources and numbers of test cases

The phishing URLs used in the test are extracted from spam emails and collected from the web using a crawler. A minimum of 100 phishing sites will be used, but possibly many more, depending on the duration of the test. For false-positive testing, at least 100 legitimate online banking websites are used.

Test procedure for browsers/security products

Phishing websites have very short lives and may be taken down only hours after they are put online. To ensure that as many phishing pages as possible can be tested while they are still active, we test all the phishing URLs we receive immediately; any that turn out to be inappropriate are excluded from the results. A URL may turn out to be unsuitable if it is not a genuine phishing site, is offline, is evidently a duplicate, or can be seen to be malfunctioning (e.g. error messages appear when the page is opened). Our automated test procedure feeds the test PCs one phishing URL, to which all machines then browse simultaneously (ensuring that the availability of the page is the same for all products).  This is done in a way which replicates a user clicking on a link in real life (as opposed to an argument being passed directly to the browser). Screenshots are taken to indicate whether or not a phishing page has been blocked and/or a warning message displayed. Each test PC is then rebooted and reset to its original configuration before the next test case begins. This ensures a level playing field for each test case, and prevents any possible confusion between warning messages for one test case and those for a following test case.

To be deemed successful, a product must warn the user that a site is considered unsafe before any sensitive information is entered or submitted. Protection mechanisms that trigger only after data entry or submission do not count as successful protection for the purposes of this test.

False positives

As with many of our other tests, we check that products are not reaching high detection rates at the expense of a high rate of false positives. A false-positives test is carried out using a number of popular legitimate websites that ask for user credentials or personal information; there will be an emphasis on online banking sites worldwide. A single false positive from an online banking site is sufficient to downgrade a program’s rating.

Summary

Related Posts