Solved

Track if sent email has been opened

Posted on 2013-01-11
31
935 Views
Last Modified: 2013-01-14
Hi,

I'm trying to figure out a way to track if a sent email has been opened.
This could done with a tracking pixel, but i don't know how it works.
The emails that are sent are customized emails that load external images from a remote server.

What would be a good way to do this? I believe some kind of fairly easy trick to accomplish this exists, so i don't want to use any external software, i'd rather build it myself.

Thanks!
0
Comment
Question by:peps03
  • 9
  • 7
  • 4
  • +6
31 Comments
 
LVL 82

Assisted Solution

by:leakim971
leakim971 earned 150 total points
ID: 38767495
<img src="path/to/images/loadImage.php?emailID=123456" />

but today messaging client ask the user to cirm they want to download image
0
 
LVL 37

Expert Comment

by:Neil Russell
ID: 38767536
As stated, most modern clients will not automatically open images.  There is NO foolproof method that will confirm OR deny that an email has been read.

The commenest aproach if you MUST confirm read of an email is to have the email content stored on a web site somewhere and just email a link to the end user that allows them to go read the content. Once they hit your website, you know that they have landed on the email.
0
 
LVL 53

Expert Comment

by:COBOLdinosaur
ID: 38767553
<img src="path/to/images/loadImage.php?emailID=123456" />

If I was sent an email with that in it my filtering would strip it, and I would log an advisory to block the ip of both the email server and the domain where the image is hosted.

You are treading on thin ice when you try to track user activity on their own computer.

Cd&
0
 
LVL 34

Assisted Solution

by:gr8gonzo
gr8gonzo earned 200 total points
ID: 38767570
I know you said you want to build it yourself, but I've built mailing engines before and there is more to it than you probably think. I am the ultimate DIY type of guy and have rebuilt every wheel imagineable, so let me pass on this advice - there is a lot of value to be gained from using a 3rd party such as ConstantContact here.

Now, that said, I know I would never be fulfilled as a DIY person without actually knowing the technique, so here it is:

A tracking pixel needs to be on the remote server. leakim is ALMOST there, but he forgot that you need the full host. With a relative path like path/to/images, the mail client is going to look locally for the image, so your remote host will never be "aware" of the image loading.

So the tracking pixel would actually be something like:

<img src="http://yourserver.com/path/to/images/loadImage.php?emailID=123456" />

In your loadImage.php script, you would take that emailID (which would identify that specific recipient) and then be able to know that your recipient saw the message (otherwise the pixel would not have loaded on its own).

The caveat here is that tracking pixels are a dying and nearly-dead art. The vast majority of mail clients will NOT load ANY remote images because spammers ruined it for everyone. Spammers began to buy lists of unsolicited email addresses and then used remote tracking pixels to figure out which ones were active so they could spam more. Remote images could also carry certain viruses and exploits, so mail clients simply started preventing the images from downloading by default.

Most will prompt the user (subtly) if they want to load images from the remote server. If the user actively clicks on the button to allow this, then the tracking pixel loads and you get notified. If the user is just lazy and has no reason to do an extra click to read the message (which is almost every person), then they won't do it, and you'll end up with a far lower "opened" rate.

===> Bad information is WORSE than NO information. <===

You can't use bad information for anything reliable. You can't base any reliable goals on bad information, and you can't reliably approximate anything, either.

The better technique for effective mailing is to give the user a reason to either load remote images or click on a link to read the message. I regularly click the button to load images in my NewEgg newsletter because I like to see the specials that they have running, and the information is contained in an attractive, readable way INSIDE the images themselves. I know they are tracking the opens, and that's all good with me.

Or if I get a message from a trusted source that tells me I need to click on a link to read the message (e.g. a self-verified email from my bank), I will do that. I would not click on a link to read a marketing message.

You also have to beware of going overboard with messaging contained inside the actual images on emails, because that can also be a spammer technique that will set off red flags. Third parties can help improve the "trustworthiness" of your emails so that spam filters will allow the message to get through, and people will be more inclined to hit the button to display all images.
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 38767576
I really need to start writing shorter posts. Two other people posted in the time it took me to write that last one.
0
 
LVL 82

Expert Comment

by:leakim971
ID: 38767577
@COBOLdinosaur, better?

<img src="path/to/images/123456/imageXYZ.jpg" />

Open in new window

0
 

Author Comment

by:peps03
ID: 38767660
Thanks all!
So the php file will contain a piece of code that adds +1 to the userid in the db.

Will it also contain a piece of code to display an actual image?
like a base64 type or something?
0
 
LVL 53

Expert Comment

by:COBOLdinosaur
ID: 38767762
@COBOLdinosaur, better?

The image still gets blocked and the ip of the mailserver and hosting domain are recorded.  It would not trigger an immediate advisory in the logs, but if the number of mails from the ip reach an alert level then it would trigger a blocking advisory.

Cd&
0
 
LVL 52

Assisted Solution

by:Scott Fell, EE MVE
Scott Fell,  EE MVE earned 50 total points
ID: 38767766
Tracking opens means nothing.  Track the click throughs to your site instead.  <a href="http://mysite.com/trackingpage.php?id=123">Click me</a>  Then on trackingpage.php update your database.  You can track individual links and email id's.
0
 
LVL 82

Assisted Solution

by:leakim971
leakim971 earned 150 total points
ID: 38767768
no, you just return the image
a short one :
<?php
header("Content-type: image/jpeg");
$image = imagecreatefromjpeg($_GET['img']);
imagejpeg($image);
?>

you may need to use an htaccess rules if you want to use and/or check for directory transversal to protect your server :
<img src="path/to/images/123456/imageXYZ.jpg" />
instead :
<img src="path/to/images/loadImage.php?emailID=123456" />
0
 

Author Comment

by:peps03
ID: 38767808
hmm oke.
i almost get it, i think.
would i put the path to the real image in the loadImage.php file?

and can i use this technique with every image? like the logo in the email header?
0
 
LVL 82

Assisted Solution

by:leakim971
leakim971 earned 150 total points
ID: 38767813
The image still gets blocked and the ip of the mailserver and hosting domain are recorded.  It would not trigger an immediate advisory in the logs, but if the number of mails from the ip reach an alert level then it would trigger a blocking advisory.

Hope you don't think I'm disagree with you

But, I put a random email from amazon :
http://xyz.cloudfront.net/email/12-14-12/Lead-4-reinvent.png
not sure what they do with 12-14-12 (the mail was sent the 14/12/2012)
I trust Amazon email, YES I want to see and loading the pictures of the email.
I did not receive this mail in my spam folder (orange.fr provider)
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 38767982
My own $0.02 - COBOLdinosaur's filtering sounds a little more overprotective than the typical mail user. Most users will hardly adjust the spam filtering with the slider bar in Outlook, much less have rules that process content, and most users are not behind custom mail servers with tuned server-side rules. A lot of home ISPs leave spam filtering rules wide-open and many business servers simply trust the default rules from their security appliance manufacturers. This means that the majority of server-side filtering is not changed.

That still doesn't change the fact that most people won't activate the tracking pixel.

leakim - no need to use GD to generate a blank pixel. You can use readfile() to stream the 50 or so bytes from an existing blank pixel GIF. It's much less resource-intensive.

peps03 - you only need to do it once per email.
0
 
LVL 108

Assisted Solution

by:Ray Paseur
Ray Paseur earned 50 total points
ID: 38769058
I think @padas is giving you good advice, in addition to the other good advice you've got here.  If you know that "Ray read the email" that is not nearly as good as "Ray read the email and clicked a link to your site in the email."  The former, if your emails are all sent to subscription-only clients, will get you a 25-30% open rate.  The latter will get you a 5% open rate (assuming your content is extremely well targeted).  This is about 10X better than print advertising, hence the decline of newspapers.

You might also want to know who forwarded your emails.  You can make an educated guess about this by counting the requests to the invisible pixel script.  Example: When I open the email, I send a +1 to the tracking script.  When I forward it, and someone else opens it, that someone else sends a +1 to the tracking script.  Etc.  This is far from foolproof; if I just read the email twice it has the same effect.

GMail asks me whether to show images in the email.  If I say, "no," and that is the natural condition if no changes are made, then the tracking pixel strategy fails.

Executive summary: There is no accountability in email.  Get over it.  Even the really good services like ConstantContact.com can't get it 100% right.  But they can and do ensure that your messages are delivered to the recipient and that you're not going to jail for spamming.
0
 
LVL 53

Expert Comment

by:COBOLdinosaur
ID: 38769168
Most users will hardly adjust the spam filtering with the slider bar in Outlook, much less have rules that process content, and most users are not behind custom mail servers with tuned server-side rules.

Very true. I have a dozen emails accounts and the incoming gets auto forwarded to my server and everything gets parsed with PHP before it makes it to my client.  The script has a white list in db table and I can set specific rules right down to individual email addresses.  So stuff coming from my bank or brokers allows images, because I want the graphs them sometimes include.  The point of filtering is more about getting good emails in the right folders as much as it is dumping the crap.  The important side is the whitelist to make sure you get the important stuff.

Cd&
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 82

Expert Comment

by:Dave Baldwin
ID: 38769483
Constant Contact and all of the other email services use both tracking images and 'click-thru' links that go to their servers first before being redirected to their destination.
0
 
LVL 11

Expert Comment

by:mcnute
ID: 38769640
@gr8gonzo. You posts are intimidating ;-)

Maybe it has been said already, but I've modified my apple mail client with secrets to send me an email when the reading has of my sent email has been confirmed. That is less a tracking. It requires the user to actually click on a dialogue in their email-clients, but it will tell you that your user has read the email.

So if apple mail can send along this information you can do also within your headers of the email.

For the reading confirmations:

X-Confirm-Reading-To: <address>

For delivery confirmations:

You have to add the Disposition-Notification-To header.

RFC 3798 Standards reading
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 38770298
@DaveBaldwin - Yes, many of them do that, but they also have an entire application to support data analysis, and a higher delivery rate due to proven anti-spam self-regulation. Higher delivery means increased actions from users (more clickthroughs and more tracking pixel activation).

Trying to rely on tracking pixels alone as an indicator of open rates is just not a good idea. As padas and I both said, clickthroughs is typically a better way to go (unless you happen to have content that users are going to be desperate enough to WANT to read that they'll enable the remote images).

@mcnute - Intimidating? :) I'm sorry - that's never my intention. If I ever disagree with someone, it's purely for constructive purposes. This is a learning site, so I try to include as much detail and context as possible.

That said, delivery confirmation will be kicked off as soon as the email is delivered to the user's mail server, not their mailbox. Failed delivery confirmation is a good way to weed out bad addresses from a mailing list, but not an indication of whether or not the user has actually even seen the message.

The Confirm-Reading-To works pretty well when you know the recipient and they WANT to explicitly notify you that they read it (great for business comm). For mass mailing, people tend to click the "No" button, meaning that you never get confirmation, even if they continue to read it.

The other issue is that it's harder to process the confirmations. With tracking images and click-throughs, the user is going straight to a script. Confirmation emails, on the other hand, come in the form of email messages. That means a separate process needs to download the confirmations from a mail server, process the email, and then finally perform the database changes to indicate a mailing has been read. This CAN be done, but if a user is going to click on your email message anyway, it's often better to try to drive them to a web site where you can then perform a more thorough attempt at a conversion.
0
 

Author Comment

by:peps03
ID: 38770299
Thanks all for the reactions.

I'm still not exactly sure what the php file will look like..

Will the file produce a gif image or should a 1px transparent gif image be loaded into the file?
0
 
LVL 11

Expert Comment

by:mcnute
ID: 38770350
I meant just for their length. I havn't read your comment, because it's so long ;-)
0
 
LVL 34

Accepted Solution

by:
gr8gonzo earned 200 total points
ID: 38770354
So let's say your tracking image is called:
http://www.domain.com/tracker.php

You have a transparent pixel GIF in the same directory:
http://www.domain.com/pixel.gif

You have a database of mailings, each mailing with its own ID, and you've sent out mailing ID 12345.

In the email content, you would have:
<img src="http://www.domain.com/tracker.php?ID=12345">

Inside the PHP script you would have something like this:

<?php
// Connect to database
mysql_connect("host","user","password");
mysql_select_db("db_mailings");

// Now get the ID and update the mailing
$ID = intval($_GET["ID"]);
mysql_query("UDPATE mailings_table SET read=1 WHERE ID=".$ID);

// Now display the transparent pixel
readfile("pixel.gif");
?>
0
 
LVL 82

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 50 total points
ID: 38770374
@gr8gonzo, you forgot to send the "Content-Type" header first.  From http://php.net/manual/en/function.imagegif.php

// Output the image to browser
header('Content-Type: image/gif');

Open in new window

0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 38771917
Dave's correct - sorry, I rushed through that script a bit.
0
 
LVL 34

Assisted Solution

by:gr8gonzo
gr8gonzo earned 200 total points
ID: 38771923
I should also mention that the above script is not very secure. It is just an example of the guts at work. Normally, you wouldn't only pass the ID (because then a malicious person could easily write a script that goes and downloads the same picture for every ID, which would mark all your mailings as read), but that's another issue. I just figured I should reinforce that the above script is not a complete one, but it should give you the idea of what's to be done.

And again, don't do the tracking pixel.
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 38771926
@mcnute - :( Yeah, my comments are long, but I try to make them valuable...
0
 

Author Comment

by:peps03
ID: 38772070
@gr8gonzo, thanks for the explanation.

You said your code isn't very secure, how could i secure it better?

And again, don't do the tracking pixel.
Why? It could give me some additional info.. i also track links inside the email. The combination seems better than only 1 of the 2.

so the image code in the php script will look like this:
// Now display the transparent pixel
header('Content-Type: image/gif');
readfile("pixel.gif");
0
 
LVL 34

Assisted Solution

by:gr8gonzo
gr8gonzo earned 200 total points
ID: 38772717
@peps03:

Think about a debit / ATM card. Just knowing the full number doesn't give you access to the funds in your bank account - you also have to know the PIN number. It's a two-part "puzzle" to gain access.

When you're dealing with IDs in databases, they tend to be sequential, so if you get an email with an ID of 246788, then there's a chance that someone else got 246787, and another person got 246789, and so on. So if you have a tracking pixel that simply takes that ID, then a malicious person could write a 4-line script that loops from 1 to 1000000 and accesses the tracking image for everyone:

<?php
for($i = 1; $i < 1000000; $i++)
  file_get_contents("http://www.domain.com/tracker.php?ID=$i");
?>

Presto, that malicious person has screwed with your data - you won't know who actually read the email now because it looks like they all got activated.

Now let's say that you have a second number that is somehow generated from the first, we'll call it a "Code"

ID    |    Code
123  |    63
124  |    74
125  |    85
782  |    172

The tracking pixel would look like:

http://www.domain.com/tracker.php?ID=782&Code=172

Inside the tracker.php, we will do an extra check to see if the Code for 782 is really 172. If it is, then we'll update the mailing. If not, we don't do anything.

Now the malicious user can't just write a simple script to loop through all the tracking pixels (he/she can TRY but it won't work without knowing how the Code is generated).

And that's the point - THEY don't know how the code is generated, but YOU do, since you can write just about ANY algorithm to do it. For example, the above codes are generated like this:

ID = 782
Code = 7 + 8 + 2 and add on the last digit of 2, so 17 and 2 = 172

ID = 123
Code = 1 + 2 + 3 = 6 and 3 = 63

You can do anything with the original ID to generate that Code as long as it's not immediately obvious (e.g. ID = 123, Code = 321 - you can probably guess how that works).

The beauty is that you don't have to store the code anywhere. If someone provides an ID and a Code, all you do is re-generate the Code from that ID and see if it matches the Code that they gave. If it does, then you can go ahead and do what you need to do. If it doesn't, just don't do anything further.

Here's another simple algorithm that will be hard to guess:

$Code = substr(((preg_replace("[^0-9]","",md5($ID))+1)*1024),0,4);

That will always give you a 4-digit Code for any ID. You generate the code when you send out your mailings, and you generate the code when you go to check the tracking pixel request, but you can use any algorithm you want as long as it consistently produces the same Code for the given ID.

Now, to answer the question about "why" not to do the tracking pixel - it's because the presence of a tracking pixel is more likely to set off spam filters, and it generally provides very little useful data. It also prompts the user to load remote images, which can also be a red flag for some users and make them skip over the message completely.

So by using a tracking pixel (and not having the credibility of a larger, established mailing service), you can actually damage your chances of getting your message to your end users. It is far better to using tracked links and simply have a KPI of sent, bounced, and clicks.
0
 

Author Comment

by:peps03
ID: 38772825
Thanks for your reply again!

I use 7 random generated big and small letters as id.
And the tracking is only for a few sent emails at a time.
So i think that is covered.

I still have some trouble adding +1 to the db table. It works fine when i load the url of the php file directly into the browser, but is seems to only work once from an email client.
If the images are loaded again, +1 is not added.

Could it be the php file is cached by the email client / browser or something?

Is there a way to prevent that?

AND sometimes the image is not loaded.
0
 
LVL 53

Expert Comment

by:COBOLdinosaur
ID: 38772860
You have been told repeatedly that this is unreliable.  So now you are seeing it on a mail client you control; think about it on clients you don't control.  The information you get will not be accurate no matter what you do.

Cd&
0
 

Author Comment

by:peps03
ID: 38773666
@COBOLdinosaur, don't get mad at me.. just asking.
i fixed it thought, the mail client cached the url.
0
 

Author Closing Comment

by:peps03
ID: 38773734
Thanks all for the help. Nailed it!
As the emails that are sent already contain 3 external images, i believe its no problem to add 1 more for tracking.
And, i'm also tracking the links, so combined i'll have some more info than none at all.
Happy tracking!
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

This article demonstrates how to create a simple responsive confirmation dialog with Ok and Cancel buttons using HTML, CSS, jQuery and Promises
This article explains how to prepare an HTML email signature template file containing dynamic placeholders for users' Azure AD data. Furthermore, it explains how to use this file to remotely set up a department-wide email signature policy in Office …
In this tutorial viewers will learn how to style elements, such a divs, with a "drop shadow" effect using the CSS box-shadow property Start with a normal styled element, such as a div.: In the element's style, type the box shadow property: "box-shad…
In this tutorial viewers will learn how to embed videos in a webpage using HTML5. Ensure your DOCTYPE declaration is set to HTML5: "<!DOCTYPE html>": Use the <video> tag to insert a video. Define the src as the URL of your video; this is similar to …

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now