Chap 3

kenj · November 2nd, 2009, 04:10 PM

page 69, lower gray box, line 5
has $p->connection, it should be $p->connect

Code:

    if (!$p->connection('mail.example.com'))

should be

Code:

        if (!$p->connect('mail.example.com'))

Also, I realize this is a quick snippet of code that may go
somewhere else, but if you are going to use it standalone as
I did for initial testing, then you need to include the POP3client.

Code:

include '../lib/POP3client.php';

I found this code useful for testing out some other pop3server functions
such as _list and retr.

page 77, the manage.php code in the gray box,
11th line down needs semicolon at the end.

Code:

    $pop->user(MANAGE_USER)

Should be

Code:

        $pop->user(MANAGE_USER);

page 79, the manage.php code in the gray box
The case 'HELP' block needs a break statement, otherwise
it will fall through to "default" and give you the 'unknown.txt'
file content.

Code:

  case 'HELP':
     $response_file = 'help.txt';
     
  //unknown command was received
  default:

It should have a break statement

Code:

    case 'HELP':
       $response_file = 'help.txt';
       break;
    //unknown command was received
    default:

After clearing up these minor issues, I got the programs to work.

******************

About the programs deleting mail

manage.php and individual.php are very powerful, they
will delete all the mail in the inbox when completed, so you need to
be careful not to test these programs with your website mail main account.
Set up a manage account and a list account on the popserver,
and edit the config.php file to point to these accounts.

daibhidh · December 3rd, 2009, 08:48 AM

Hi, very useful Kenj...but I'm still having problems with this script...

It all seems to run fine, but when i comes to locating subject line (in email) commands such as 'SUBSCRIBE', the variable just seems to be blank and the defualt email response (unknown command) is returned...

Same problem also when someone has subscribed (done manually via phpmyadmin to test the system)...when you send the email, the text doesn't seem to be properly captured and thus no email is sent to the list and, while a digest file is ctreated, nothing isp laced in it...

I think it's problem in and around the ' preg_match_all ' commands. I think the code is finding the correct data and placing it in the $matches variable, but the data is not then coming out of the $matches array properly...

Any ideas would be most grateful!

Daibhidh :)

kenj · December 3rd, 2009, 04:22 PM

Hi Daibhidh,

Thank you for adding to the discussion.

You may have done a lot of this troubleshooting
already, but if you can tell us what happened
when you did these things, then that would be
helpful.

In the manage.php file, it would be helpful to comment out
the line that deletes the mail. It is near the bottom,

Code:

$pop->dele($id);

This way you don't have to keep sending it mail. You can
remove the comment once this problem is resolved.

Then look in lib/config.php and see what your MANAGE_EMAIL,
MANAGE_USER, and MANAGE_PASSWORD are.
Go to some other mail system like yahoo or gmail or something
and send a letter to that MANAGE_EMAIL. Put SUBSCRIBE in the
subject line.
Then go to the mail account where MANAGE_EMAIL is and
log in as MANAGE_USER with MANAGE_PASSWORD.
Verify that the mail is there.

Now, that we know we have mail in the proper account,
we can move on to the manage.php program.

Add some echo statements in the manage.php program

Add a echo statement right after

Code:

$message = $pop->retr($id);

echo 'message is ' . $message;
Just to make sure you are getting messages. There will be a lot of
junk in there, but see if the SUBSCRIBE is in there in the subject line.

Add this echo statement after the subject trim line.
echo 'After subject trim, id is ' . $id . 'from is ' . $from . ' subject is ' . $subject;

Is the subject there?

For individual.php try similar troubleshooting.

You have probably already done a lot of this, but if
we know the results then that can help to
narrow it down.

*****

Here are my experiences.

I used the downloaded code and made those
changes I mentioned, and it was working for me.
I am using php version 5.2.6

Good luck with this. Let us know how it goes.
Thanks.

daibhidh · December 3rd, 2009, 05:53 PM

Hi Kenj...many thanks for your help...

I have debugged and checked everything down to the bit where I've added

Code:

echo 'After subject trim, id is ' . $id . 'from is ' . $from . ' subject is ' . $subject;

This code returns the following when the script is run: "After subject trim, id is 1from is subject is "

So, on this basis it looks like the problem I have is that either the preg_match_all function isn't working properly, or the subject of the email is being stored in a different part of the array...could this be the problem? The script is finding the email address ok as I tried an echo on this, it's just not finding the subject of the email.

I tried changing

Code:

$subject = trim($matches[2][1])

to

Code:

$subject = trim($matches[2][2])

and this seemed to solve the problem in terms of the subject, but it then meant that the email address var ($from), rather than having sender name AND email adddress (e.g. "Joe Soap" <[email protected]>), changed to just email address part (e.g. [email protected]).

Do you have any ideas? Maybe the code is extracting the correct parts of the email, but placing them in different parts of the array?

Many thanks for your help so far; it is much apprecaited!

Daibhidh :)

PS - My server is running PHP v5.2.11

kenj · December 3rd, 2009, 09:10 PM

Thanks for sharing this.

This is interesting.

Given the content of your mailed message the results are starting to make sense.
In the message there is no email address after the "From:", there is only the
sender's name. So that is why there is no email address. This makes sense,
even though it's not what you want.
Your message appears to be coming from Microsoft Office outlook.
Maybe the user configured it to hide his actual email address from
the "From" field.

My messages are coming from yahoo mail.

Here is a snippet from my message

Date: Thu, 3 Dec 2009 14:39:28 -0800 (PST)
From: MyfirstName MyLastname <[email protected]>
Subject: HELP testing
To: [email protected]

Notice the text string after From, it contains the
email address, so mine worked.

Just for testing purposes, you may want to set up a
yahoo email account and see how it works then.

You may have found an issue.
The preg_match_all is simply parsing over the message.
It's not clear if this can work for all types of messages,
as you have just shown us today.

Maybe someone could replace that preg_match_all call
with a function that deals with all kinds of message
formats and extracts out the proper from and subject
fields.

You have brought up some very interesting issues.
This is a good find on your part, nice work.
Thanks.

daibhidh · December 4th, 2009, 10:54 AM

Hi Kenj,

I posted this on the other forum string, but given it also follows on front your last post, I thought it best to post it here too...

---
I think I may have found a work-around for this problem...I was studying the code and it seemed to me that pre_match_all was struggling to process all of those requests at once given the diversity of message headers etc. between mail servers, so I broke them down into bite-sized pieces:

Code:

    preg_match_all('/Date: (.+)/', $message, $date_matches);
    preg_match_all('/From: (.+)/', $message, $from_matches);
    preg_match_all('/Subject: (.+)/', $message, $subject_matches);
    preg_match_all('/boundary="(.+)"/', $message, $boundary_matches);	
	
    $date = trim($date_matches[1][0]);       
    $from = trim($from_matches[1][1]);       
    $subject = trim($subject_matches[1][0]);    
    $boundary = (isset($boundary_matches[1][0])) ? $boundary_matches[1][0] : false;

...now it seems to work for all emails I send, processed through individual.php, no matter what account (e.g. hotmail, yahoo) they comes from...

Will try this for the digest file too...

Also, your fix for the manage.php file worked fine until I tried my work email, which is yet another mail server. It returned an error everytime I tried to subscribe. I change the code along the lines of the above, and now it seems to work without any problems...

Code:

    preg_match_all('/From: (.+)/i', $message, $from_matches);
    preg_match_all('/Subject: (.+)/i', $message, $subject_matches);
    $from = trim($from_matches[1][1]);
    $subject = trim($subject_matches[1][0]);

Do you think this is all just beginner's luck, or is it a proper fix?! :)

Daibhidh
---

I've been testing the revised code all morning with a variety of email addresses and it seems to work fine. The digest file also works fine now too...

However, in testing, I've come across a problem when people send an email to the list which has an attachment attached. I read in the book that this script wasn't built to deal with attachments, so perhaps stupidly thought that it would simply ingnore any attached file.

What I've found is that the file, in its base64 encoded state simply over-writes the content of the email and what's sent to the list is just a long jumble of meaningless (to non-comuters) text.

I'm guessing that this might be something to do with the fact that an attachment seems tp throw up and additional 'boundary' tag which is picked-up by the preg_match_all function...this in turn shuffles the position of the data in the array and returns the coded attachment file with the email, rather than the email message text...

This isn't very useful unless you can put a block on attachments being sent to the list...which it's possible in my situation.

So, I was wondering there's any way to screen attachments and parse them from the eventual email, or even better (but no doubt tricky) to deal with attachments and allow them to be processed by the script and then re-attached to outgoing list emails...

Any ideas?!

Getting there, but it's slow work!

Daibhidh :)

kenj · December 7th, 2009, 05:10 PM

Hi Daibhid,

The number of matches for the first pattern will affect the index of the
two dimensional array for all the subsequent patterns.
That is what happened the other day when the "Subject" pattern got pushed
from [2][1] to [2][2].
Breaking it up as you have done eliminates this problem.

Basically the preg_match_all expects one "From: "
and it expects one "Subject: "

If the mail header has more than one match for either of these,
then we need to deal with that.

My fix was a fix to "not match" on the "X-Envelope-From: " pattern.
This made it work for Outlook and Yahoo samples.
It also helps eliminate this patten match on any other mail
formats that contain the "X-Envelope-From: " header.

Now, your work email may have a format that holds yet another
extra "From: " or "Subject: " pattern that we need to work on.
However, there is something else I have a question about.
The book gets its "From: " from [1][0], and you are now
getting it from [1][1] (in manage.php) Getting it from [1][1]
would work if you had an extra From: like the
X-envelope-From header, but would not work if you didn't
have it.
Also, I did not see the regular expression [^-]
in front of the "From", so "X-envelope-From: " would match the
pattern.
Try adding the regular expression [^-] in front of the From like
mentioned in the last post, and change the index back down to [1][0],
like it is in the book.

That regular expression [^-] is left bracket, caret, hypen, right bracket.

Thank you for getting the different kinds of mail formats,
that is really very helpful on the testing.

daibhidh · December 7th, 2009, 05:26 PM

Hi Kenj,

I have tried emailing the script from about 10 different email addresses now, all on different email servers. They all come with headers as per my original outlook email...

As I posted in an earlier post, I've had problems in the past with a 'malformed header' error when using SquirrelMail with the same receiving email host. I'm wodering whether the problem I have revolves around the email server that the receiving email boxes are hosted on. If this is the case, then I suspect, certainly from the various test emails I've sent, my fix may well work for all emails sent to the email host I use for the script; however may not work if I transfer to an email server which doesn't return these somewhat strange headers...do you think I could be right?

At present, the email server I'm using is on my 'test' host. I plan to switch everything to a different host and a different server once it's all up and running. I suspect that this may mean that the script will run fine as shown in the book once the move is complete...once I've done this move to a new server this week, I'll post my results back on this forum...maybe it'll explain things?

Either way, it perhaps shows that the script doesn't work 100% in all email environments and may need tweaking depending on what headers your email host spits out?

What do you think?! :)

Daibhidh :)

kenj · December 7th, 2009, 07:23 PM

Hi Daibhidh,

You have really been testing a lot, that's good.

I have not tried this with SquirrelMail. Keep in mind that this
Chapter 3 is using POP3 protocol. If you go out to the
squirrelmail.org/wiki it says it needs a Mail Fetch plugin
to receive mail from a POP3 server. Maybe you already have
this set up, but if not, it might be something to take a
look at.

You probably already know this, but I'll state it anyway.
The input to manage.php comes from this statement.

Code:

  $message = $pop->retr($id);

so if any $message does not look right, you would need to go
back and look at what is going on with your mail servers.

If each $message does look right, then we would need to do more
work parsing the mail formats to extract what we need.

I use a mail server that my ISP has and that has been working
well for me.

daibhidh · December 8th, 2009, 07:46 AM

Hi Kenj,

I have placed [^-] in front of 'From' as mentioned in your post and changed the index back down to [1][0], like it is in the book...this works for me now, so long as I use seperate preg-match code for each querey...just doesn't seem to want to work when it's all bundled up together in one multiple preg-match command...however, whether or not I used [^-] in front of 'Subject', it always ends up in [1][0] as well, rather than [2][1] as shown in the book...strange...any thoughts?

I have this same problem in the individual.php file where preg_match returns:

"Array ( [0] => Array ( [0] => Date: Tue, 8 Dec 2009 11:27:06 -0000 ) [1] => Array ( [0] => Tue, 8 Dec 2009 11:27:06 -0000 ) )

Array ( [0] => Array ( [0] => From: "Daibhidh Baldwin" ) [1] => Array ( [0] => "Daibhidh Baldwin" ) )

Array ( [0] => Array ( [0] => Subject: test ) [1] => Array ( [0] => test ) )

Array ( [0] => Array ( [0] => boundary="----=_NextPart_000_0057_01CA77F9.6196B830" ) [1] => Array ( [0] => ----=_NextPart_000_0057_01CA77F9.6196B830 ) ) "

So, I have adapted the code as follows:

Code:

    $date = trim($date_matches[1][0]);       
    $from = trim($from_matches[1][0]);       
    $subject = trim($subject_matches[1][0]);    
    $boundary = (isset($boundary_matches[1][0])) ? $boundary_matches[1][0] : false;

Now, I've tried and tested this using loads of email accounts and it works fine with that slighlty different code above, but doesn't work at all with the code in the book...very strange, but the 'fix' works so it's not a problem for me just now...

The above aside, I have also come across a few other problems. I'm not sure if these are to do with the above problems or if they're unrelated...

1) If you send a message to the list from hotmail/yahoo etc. then the message sent round to list members is a bit malformed - this problem does not occur if the email is sent from outlook or squirrelmail. The email looks like this:

"Afternoon all=2C

We have to transfer the club website and email functions to a new server given limited bandwidth and functionality on the current package. This will be happening over the next day or so...

This is a quote: "quote test here in inverted commas"

This is an apostrophe test: it's my house=2C not Claire's

Bla bla laaa

Many thanks=2C

Daibhidh
=20"

=20 seems to replace carriage returns, =2C replaces a comma, while not shown in the example above, Daibhidh inverted commas occasionall get replaced by =93 and =94 and apostrohpes are also replaced by weird code...do you know why this happens?

2) If you send an email with an attachment to the list, then the encoded attachment takes the place of the email message and is sent round the list. I know the book says the scipt doesn't handle attachments, but I assumed it would ignore them, not over-write the email message with the raw code!

I have written a partial work-around for this which seperates out the attachment code via another preg-match boundary qeurey. It basically then adds the attachment boundary to the '$chunks' process for getting rid of before the email is forwarded to the list. Seems to work for all attachment types, with the exception of text files...

This added to preg_match code:

Code:

$attachment = (isset($boundary_matches[1][1])) ? $boundary_matches[1][1] : false;

This added to the multipart messages section of code (after the original 'chunks' section...

Code:

// split the message into chunks2 to remove attachment data
$chunks2 = preg_split('/' . $attachment . '/', $message);
				
array_shift($chunks2); // drop headers before MIME boundary
array_shift($chunks2); // again to drop headers after MIME boundary
array_pop($chunks2);   // drop trailing --
								
// merge chunks 1 and 2 ready for processing
$chunks = array_merge((array)$chunks, (array)$chunks2);

Seems to work for me, though looking further into why it doesn't work with text file attachments...of course, the best solution would be if I could make this script handle and forward attachments...is that easy to do?

Sorry to ask so many questions...just as one problem appears to be sovled, another crops up!

Many thanks,

David :)