How reliable is HTTP_REFERER to use in mod_rewrite?

Posted on 2011-10-10
Last Modified: 2012-05-12
I notice that a lot mod_rewrite rules are based on the value in HTTP_REFERER but can this be trusted?  Don't most browsers have a Privacy mode now that leaves HTTP_REFERER empty?  And can't an attacker easily modify HTTP_REFERER?

It seems that legitimate users browsing privately would get blocked if the rule was looking for something special in HTTP_REFERER, and illegitimate users could put whatever you're looking for in there.
Question by:steva
    LVL 34

    Accepted Solution

    1. Just about any HTTP header can be modified manually by an attacker, including HTTP_REFERER. However, it's typically not efficient for someone to fake a referer. If someone else wanted to leech one of your resources (e.g. an image), they'd have to do it server-side (e.g. a script that goes and pulls the image and passes in a fake referer), which takes up MORE resources to do than to simply serve up a copy of that same resource.

    2. The privacy mode effects depend on the browser and configuration, but to be blunt, I think the Privacy mode is most often used for the sake of surfing adult content in a way that does not leave behind cache files and cookies once the session is finished. The second-most used reason is for surfing secure content on a public computer, such as a shared library PC. You need to ask yourself if either scenario fits the purpose of your site and user base, if you want to know if privacy mode will actually affect you.

    3. I've been a web developer since the beginnings of the web that we know today, and less than 1% of all my traffic across all of the sites that I either own or maintain (or have maintained in the past) is traffic that did not have a HTTP_REFERER enabled. While it's statistically possible that I have a bad sample, the fact that it's enabled by default on people's PCs and rather difficult for most people to disable, means that you can count on it being enabled for most of your traffic. I think you have to worry about strange/unknown browsers and their compatibility before you start worrying about HTTP_REFERER being unavailable. In other words, don't worry about it. It's TECHNICALLY optional, but it's always there when you're following a link.

    All of this said, HTTP_REFERER is the "light" kind of security. It's extremely easy to bypass if someone really wants to do it. It's just meant to discourage the majority of people who will simply link to an image, for example. It should never be used as a measure in a scenario where security actually matters (e.g. a login screen).

    However, you should be able to use it reliably in mod_rewrite rules.

    Author Comment

    Thank you!

    Featured Post

    PRTG Network Monitor: Intuitive Network Monitoring

    Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

    Join & Write a Comment

    As Wikipedia explains 'robots.txt' as -- the robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a websit…
    If your site has a few sections that need to be secure when data is transmitted between the server and local computer, such as a /order/ section for ordering or /customer/ which contains customer data, etc it would of course be recommended to secure…
    In this sixth video of the Xpdf series, we discuss and demonstrate the PDFtoPNG utility, which converts a multi-page PDF file to separate color, grayscale, or monochrome PNG files, creating one PNG file for each page in the PDF. It does this via a c…
    Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…

    728 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    19 Experts available now in Live!

    Get 1:1 Help Now