p2p.wrox.com Forums

p2p.wrox.com Forums (http://p2p.wrox.com/)
-   Forum and Wrox.com Feedback (http://p2p.wrox.com/forum-wrox-com-feedback-56/)
-   -   Reply E-mail System Requirements (http://p2p.wrox.com/forum-wrox-com-feedback/644-reply-e-mail-system-requirements.html)

Hal Levy June 17th, 2003 11:03 AM

Reply E-mail System Requirements
 
Requirements are not negotiable.. We need to meet these completely to implement an e-mail solution.

1. Users posting messages must be validated against the subscription database. Validation must use some kind of information OTHER than the FROM: address. This means a posting password or some kind of digital signature to prevent against e-mail spoofing.

2. Headers that are "extraneous" must be stripped from postings.

3. Detection and removal of "overquoting" should be pretty foolproof.

4. The system must resist spam well.

5. The system must recognize out of office replies and derail storms of these.

6. Some companies send delivery notices for every e-mail received- these must also be prevented from getting to the list.

7. The system must intelligently handle bounces

8 The system must integrate with existing mail systems (MSSMTP or Lyris SMTP)

The system runs on IIS/SQL2K. The machines are clustered. Running ASP 3.0 (.NET is ok, with justification)



Hal Levy
Daddyshome, LLC
NOT a Wiley/Wrox Employee

Jeff Mason June 17th, 2003 01:47 PM

Boy, I don't know, Hal.

Starting off by stating that the requirements are not negotiable doesn't exactly present a tone conducive to constructive comments. If there is no possibility of give-and-take, what's the point of commenting?

I'll comment anyway. :D
Quote:

quote:1. Users posting messages must be validated against the subscription database. Validation must use some kind of information OTHER than the FROM: address. This means a posting password or some kind of digital signature to prevent against e-mail spoofing.
I'd be very curious to know what you envision as a workable implementation of this. I certainly agree that only registered users should be allowed to post to the forum via email. I can't see, though, a workable way to implement "...a posting password or some kind of digital signature ..." and still allow me to use the reply feature of my email client.

I only want to interact with the forum via email for day-to-day message posting and response to forums I have subscribed to. I'll go to the web page now and then for profile maintenance or to poke around in forums I have not subscribed to, but for the most part I want to use email as the primary means of communication with the forum. I want to receive a posted forum message via email. When I decide to respond to that message, from within my email client I simply want to click reply, quote the original as appropriate, add my comments and click send. If I want to start a new topic, I simply want to send a message to a specific email address and have the subject of my email be the topic subject.

If I have to add a password or other identifier, how would you propose I do it? Manually add another line to my post, or somehow insert a custom email X-header? (Can you even do that with Outlook? - too many of us use Outlook I fear) I don't see manually adding a password or digital signature line as a workable solution. I'll forget 50% of the time, if not more. Adding custom code to the email client won't work, either, as there are too many different clients in use, and no doubt some corporate rules would preclude 3rd party modification of a user's email client.

I just want to respond to the post, and if I have to go through hoops to do so, I won't, or more likely I'll simply forget - the email will (presumably) bounce, and I'll just give up.

I still would like to see evidence or testimony that email spoofing ever was a problem on the old email list.

I think Wiley (and now, you) is blowing this potential problem all out of proportion.
Quote:

quote:2. Headers that are "extraneous" must be stripped from postings
What's your definition of an 'extraneous' header? You are talking about email headers aren't you? They don't display anyway, so who cares?
Quote:

quote:3. Detection and removal of "overquoting" should be pretty foolproof.
I think limiting quoting to only the immediately prior message is appropriate. How you'll detect that I don't know, given various quoting styles (top versus bottom quoting), HTML vs plain text, etc.
Quote:

quote:4. The system must resist spam well.
What's spam? Nobody's figured out a foolproof or even workable way to detect it. If you have, what are you doing here? - go make a zillion dollars selling your solution ;)
Quote:

quote:5. The system must recognize out of office replies and derail storms of these.
Out-of-office replies are indeed annoying. I welcome any way you can to suppress them. I've never seen a "storm".
Quote:

quote:6. Some companies send delivery notices for every e-mail received- these must also be prevented from getting to the list
I've never received such a thing - I'm not sure what you are talking about here. Do you mean that some places actually acknowledge an email that I send to one of their addresses? Why would anybody do that?
Quote:

quote:7. The system must intelligently handle bounces
Bounces should be dropped on the floor, and perhaps the user to whom the message is sent should be automatically disabled from being sent any more email after a few (very few) such bounces.
Quote:

quote:8. The system must integrate with existing mail systems (MSSMTP or Lyris SMTP)
List managers already integrate nicely with existing email systems, you know... Tell me again why we are reinventing the wheel here?

Jeff Mason
Custom Apps, Inc.
www.custom-apps.com

Dan Jallits June 17th, 2003 05:13 PM

maybe it is late in the day for me Jeff (16:08 CST) or I haven't had my usually mix of Venti whatever's and Code Red's, but that last post seemed pretty sarcastic. Please disregard if I am wrong, but let's try to keep this a nice place

Best Regards,
Dan Jallits

Jeff Mason June 17th, 2003 05:38 PM

If my post comes across as sarcastic, I apologize to the community and especially to Hal. That was certainly not my intent at all.

I honestly don't understand some of the requirements, and I honestly don't know how some of them might be met.

I'm truly sorry if Hal, you, or anyone else interprets my questions and concerns as sarcasm; they certainly were not intended to be.



Jeff Mason
Custom Apps, Inc.
www.custom-apps.com

Hal Levy June 17th, 2003 07:30 PM

Jeff,

I am repeating what I have been told. Wiley is unwilling to have an e-mail system that does not meet ALL of these requirements and I am told these are not negotiable. I have collected their requirements and posted them here. I am not responsible for, nor do I necessarily agree with any or all of the requirements.


To answer your specific questions:

1. I don't know how we could implement the security the way Wiley is requiring it. Adding a line with a password perhaps. Or a PGP signature both would work, however are very invasive. I am looking for ideas from the community on how to do it. Wiley is not concerned with if it was a problem in the past- (this is what I mean by not negotiable) they want this functionality.

2. Yes, headers do appear in the e-mail when it's posted on the web site- we must filter them so they don't appear on the web boards. The web boards are not going away.

3. Exactly. I need ideas from the community.

4. I said resist- it doesn't have to be foolproof. But it needs to be able to do a fairly decent job of it.

5. Classic P2P had ACTIVE moderation- that's how come you never saw the storms (or the spam). They were blocked by the moderator(s). A storm is caused when Out of Office replies respond to Out of Office replies .. And so on and so forth...

6. I am told that many .GOV sites reply with a "receipt" ack for every message that comes into the domain. I have not experienced this- however Wiley is concerned about the problem.

7. Yes, and how is all that done? We are building a system from scratch here :)

8. Because the list managers don't meet all the requirements (1-7) plus the requirements for the web package. You and I may not care about the web interface- however, hit's on the old P2P show that it WAS used quite frequently.



Hal Levy
Daddyshome, LLC
NOT a Wiley/Wrox Employee

KenSchaefer June 18th, 2003 04:05 AM

Hi Hal,
WRT to point 5 - there are very few mail servers these days that result in "OOO" storms. Even everybody's favourite whipping post, MS Exchange, only sends 1 OOO to each "from:" address, and generally not to the list address (but rather the from: address).

WRT to the last comment "list managers" don't meet all the requirements, I'm interested to know which ones don't...

My experience of both Lyris and LSoft was that:
1) you could configure it so that each person had to use a password to confirm each message -or- an admin could approve each message

2) taken care of automatically

3) doesn't do this - whatever the user posts is included, but moderators can edit user posts

4) Both resist spam well, even if you just force the "from:" to be from a list members registered email address (coupled with moderation would eliminate all spam)

5) Addressed above

6) Those people should be unsubscribed from the list

7) Both LSoft and Lyris do this well. Lyris allows "x" number of bounces in "y" days before unsubscribing someone. Bounces never make it to the lists.

Lastly, Lyris can run off an SQL Server database, allowing you to build you own custom *web interface* to the list stuff, which is probably easier than trying to reinvent the wheel with-the-respect to building a listserver system... :-)

Cheers
Ken

www.adOpenStatic.com

KenSchaefer June 18th, 2003 04:06 AM

I should just clarify my previous post. I was just trying to say that both LSoft and Lyris pretty much address the stated concerns, and if you really want to build a similar system, it's probably best off looking at how those two systems work as a starting point.

Cheers
Ken

www.adOpenStatic.com

Hal Levy June 18th, 2003 08:04 AM

Everyone,

As I said, This isn't negotiable from the Wiley standpoint.

Arguing why LSOFT or Lyris will meet their needs isn't going to get us anywhere. I have been told that they will not consider anything but a MOD to Snitz.

Yes, I agree that it would be *MUCH* easier to implement a web front end on Lyris than an e-mail system on Snitz. But this is what we have been given to work with. And- again- I was told Wiley refuses to consider changing to Lyris and developing a web interface.


Hal Levy
Daddyshome, LLC
NOT a Wiley/Wrox Employee

JSample June 18th, 2003 09:49 AM

I will offer a “very brief” reason for each of the constraints that we posed to Hal. You don’t have to agree that they are good reasons, but in our mind they are valid.

1. SPAM was not an issue on the old site due to active moderation. Wiley does not have the dedicated resources to moderate a site that is generating hundreds postings a day after only a few weeks. With strong authentication we retain the ability to ban offenders, and make sure that if a user says something on the forum that there is no room for claiming someone else spoofed their account. For the future of the site we feel that SPAM is an ever increasing threat an needs to be addressed now rather than after it becomes a problem for everyone on the site.

2. Whether you have been affected yet or not, SPAMMERS are using address spoofing more and more each day. In order to not aid spammers in anyway, we keep everyone’s e-mail address on the site a secret so they cannot be harvested. If you reply through e-mail and the header information has your e-mail address in it, then we are just inviting SPAMMERS to come and take your addresses for sending their messages later.

3. Overquoting is a minor annoyance in e-mail, but on a web forum it makes the messages almost un-readable. While there are many of you that would only use email given the opportunity, there are an equal amount that prefer to only use the website and keep their mailboxes clutter free. We need to make sure that in fixing the site for one group we don’t turn around and break it for the other.

4. I believe that constraint number one ensures that the site will be very SPAM resistant. Also when you and number 2, this site become a model for what every SPAMMER on the internet DOESN’T want to see.

5. Obviously this is a threat to everyone using the system, but also one that every list based system has solved for. Their solutions should be easily replicated.

6. As funny as it may sound this is a new “policy” that many .GOV and legal sites have adopted. Wiley sends out almost a million newsletters each week from etips.dummies.com and our Cliffnotes etips. We are seeing more and more replies that simply state “your email has been successfully delivered to someone@somewhere.gov” Obviously no one wants to see the forums or their mailboxes cluttered with these.

7. This is a fairly simple requirement but with the registration system requiring valid email address this should not be a major problem on the site.

8 This one is a simple support issue. MSSMTP and Lyris are the technologies that we currently support and have already invested in. If a solution is found that uses sendmail for example, we would have a problem supporting it properly.

I hope you can now see why we have made these constraints and hopefully assist Hal in finding a resolution to them.

Thank you,

James Sample
Director, IT-Infrastructure
Wiley Publishing, Inc.

David Cameron June 18th, 2003 08:16 PM

I'm going to jump in on this late.

Quote:

quote:3. Overquoting is a minor annoyance in e-mail, but on a web forum it makes the messages almost un-readable. While there are many of you that would only use email given the opportunity, there are an equal amount that prefer to only use the website and keep their mailboxes clutter free. We need to make sure that in fixing the site for one group we don’t turn around and break it for the other.
Can I say at this point that you have broken the site for one group already, those people who used the site as a mailing list.

Ss I see it is that Wiley is concerned mostly with the web interface. I was concerned with the mailing list side of it. From my viewpoint improving on the original wrox web interface is a good thing, but not at the cost of destroying the email interface.

Secondly I'd it appears that wiley is being quite inflexible. Solutions have been proposed (more than once) for the problems that have been suggested, and yet the same problems continue to get put forward as reasons for not moving to an email reply system. Take point 2 above. Ken has made the point that the lyris system removes these. Others have suggested regular expressions. ASP 3.0 supports regular expression search and replacement, so use that remove all email addresses in postings that are not enclosed in ]URL[.

The issue as I see it is that Wiley has made a decision not to provide an email interface. Period.

regards
David Cameron

Hal Levy June 18th, 2003 08:29 PM

David,

That is not correct. I have the requirements and if I can present a solution that will meet those requirements - without an overwhelming cost- I have been told that an e-mail interface will be provided.

If you read James' posting you will see he was explaining each of the requirements- he was not saying a solution wasn't found for them yet.

Yes, We have a solution for the e-mail address problem- we can run each message through a replacement routine that strips out e-mail addresses. Great.

7 More to go...


Hal Levy
Daddyshome, LLC
NOT a Wiley/Wrox Employee

David Cameron June 18th, 2003 08:43 PM

My apologies, I read that as a list of outstanding problems.

regards
David Cameron

David Cameron June 18th, 2003 10:27 PM

Ok then, possible solutions:

1. Maybe by IP address. Most people will post from a limited range of IP addresses. If I'm posting from work there is just one IP address, if from home it is a little more difficult. Still most people will get their dialup IP address from a limited pool. The problem with this solution is how to find that IP address range. One possiblility may be to allow those people who *can* specify a few IP addresses to post be email. Also each time a person logs into the site, you could record that IP address and offer to set that as the email IP address if different from the current email IP address.

2. AFAIK headers follow a standard format, <name>: value. In email postings you could also get a number of >> in front of it. The nubmer of headers used can't be huge, so why can't generate a regexpr based on the known headers and remove them. eg ">{0:10}(From: |To: ).*\n". Furthermore you could update your list of headers from the incoming emails, which will give you a list of all the headers that are being used by differene email clients (and people using the email clients). This may not be a 100% solution, but should be worth a try. The worst problem that I can see is the chance of someone posting some text by email that looked like a header and the text being deleted.

3. Clear any lines that are prefaced by more than 2 >, and compare the earlier messages. If you get a match to a line in an earlier message, delete it.

4. Can't comment, but limiting posters to those who have accounts and if those email addresses are not shown on the site this should have some effect. Also if suggestion 1 is followed this should be less of an issue. The only danger is spammers signing up for accounts. Still I guess that is already an issue. If I cared enough I could write a perl script to automatically create a new account, login and post a message to all forums.

5. Don't know enough about the issue, but surely Out of Office messages generally conform to a pattern.

6. Ditto as above. Possibly for this and above you could add a button to the page to set the message as a new format that has slipped through. That way your filter can be "trained" with little work on your part.

7. Ditto as above + comments by Jeff.

I don't think that there will be a perfect solution, but there may be one that comes close to fitting the requirements.

regards
David Cameron

David Cameron June 18th, 2003 10:38 PM

I should clarify 3. Any lines that have more than 2 >> are assumed to be old quotes. One problem with old quotes is that as the > build up, more carriage returns get added and the lines get split up and harder to recogise as earlier quotes. Otherwise if any lines are prefixed with 1 or more > then they can be checked against earlier messages.

Note that this should be applied to the messages that are posted to the site *not* the messages that get emailed out. Reason being, as Ken said, the quotes make it a lot easier to thread the conversation back.

regards
David Cameron

KenSchaefer June 18th, 2003 10:44 PM

Hi Hal,

I wasn't trying to say that we should use Lyris or LSoft. I'm saying that these products already do most of what you need, so, we should look at how *they* handle these issues and replicate them, rather than rebuilding the wheel. I'm pretty sure you can still get evaluation editions of both of those two products.

Cheers
Ken

www.adOpenStatic.com

David Cameron June 18th, 2003 10:46 PM

And the RegExpr in suggestion 2 is wrong:
">*?(From: |To: ).*\n"

regards
David Cameron

JSample June 19th, 2003 07:13 AM

David:

I'm sorry for the mis-understanding. Let me state for the record that I REALLY want to implement an email reply system. There were just some questions on why we put put those 8 constraints on Hal and for good or bad I wanted everyone to know our thought process.

As for Ken's post I have to agree with him that Lyris especially has solved most of the issues above, and looking to them for inspiration as a terrific idea.

As I've stated to Hal before, if there is any information I can provide or anything I can do to help on this project please let me know. I am not a programmer (stupid hardware guy[:I]) but I will do what ever I can to help.

Thank you

James Sample
Director, IT-Infrastructure
Wiley Publishing, Inc.

Hal Levy June 19th, 2003 09:42 AM

Quote:

quote:Originally posted by David Cameron
1. Maybe by IP address. Most people will post from a limited range of IP addresses. If I'm posting from work there is just one IP address, if from home it is a little more difficult. Still most people will get their dialup IP address from a limited pool. The problem with this solution is how to find that IP address range. One possiblility may be to allow those people who *can* specify a few IP addresses to post be email. Also each time a person logs into the site, you could record that IP address and offer to set that as the email IP address if different from the current email IP address.
You would rather connect to the web site to validate your IP address than use PGP to sign the e-mail that your sending? I was also thinking that perhaps we could have e-mail that is not identified attached to the user account sending it- and requiring a quick login to the web site to confirm the posting (or generate an e-mail to the poster to let them confirm via e-mail). Kind of self-moderation.


Quote:

quote:
2. AFAIK headers follow a standard format, <name>: value. In email postings you could also get a number of >> in front of it. The nubmer of headers used can't be huge, so why can't generate a regexpr based on the known headers and remove them. eg ">{0:10}(From: |To: ).*\n". Furthermore you could update your list of headers from the incoming emails, which will give you a list of all the headers that are being used by differene email clients (and people using the email clients). This may not be a 100% solution, but should be worth a try. The worst problem that I can see is the chance of someone posting some text by email that looked like a header and the text being deleted.
Yes, headers should have a pretty standard format and we should be able to catch most of them with a regex.

Quote:

quote:
3. Clear any lines that are prefaced by more than 2 >, and compare the earlier messages. If you get a match to a line in an earlier message, delete it.
As you said in another message- this is complex because of line breaking. We also need to deal with outlooks love of the "indented" reply and filtering that as appropriate. For that matter, we need to figure out how to filter all the HTML garbage that Outlook (and others) add to the e-mail.

Quote:

quote:4. Can't comment, but limiting posters to those who have accounts and if those email addresses are not shown on the site this should have some effect. Also if suggestion 1 is followed this should be less of an issue. The only danger is spammers signing up for accounts. Still I guess that is already an issue. If I cared enough I could write a perl script to automatically create a new account, login and post a message to all forums.
I agree- #1 covers this greatly unless a spammer signs up for an account. I would say the best way to prevent that is to use the system used elsewhere - create a graphic that needs to be human-read and entered into the form to create an account.


Quote:

quote:5. Don't know enough about the issue, but surely Out of Office messages generally conform to a pattern.
Honestly, I have no idea if they do or don't someone want to investigate this?

 
Quote:

quote:6. Ditto as above. Possibly for this and above you could add a button to the page to set the message as a new format that has slipped through. That way your filter can be "trained" with little work on your part.
Quote:


7. Ditto as above + comments by Jeff.
I agree that 4, 5, 6, and 7 merge together into "Filtering Unwanted messages"
I agree that problem #4 goes away If we implement #1 well.

I'd really like to come up with a way to meet Requirement #1 in a way that everyone is happy.





Hal Levy
Daddyshome, LLC
NOT a Wiley/Wrox Employee

David Cameron June 19th, 2003 07:54 PM

James:

I should be aplogising, I jumped in on this one late and wasn't up with some of the other feedback at the time. Sorry.

Hal:

Quote:

quote:You would rather connect to the web site to validate your IP address than use PGP to sign the e-mail that your sending? I was also thinking that perhaps we could have e-mail that is not identified attached to the user account sending it- and requiring a quick login to the web site to confirm the posting (or generate an e-mail to the poster to let them confirm via e-mail). Kind of self-moderation.
Yes. In my case I mostly post from work which has a static IP address. Also I tend to post a bunch of messages at once. So even if I were posting from home or somewhere else it would involve setting the IP address once. From then on I could just reply to emails. The disadvantage of signed pgp messages is that I'd need to remember to sign each message I send. Basically I am keen for something I set once (or once per day) and forget about.

Your suggestion would also work well, although it would mean that you couldn't really be part of a conversation unless you cleared the messages frequently each day. It would still be a great improvement.

Another thought, is it possible to have multiple authentication methods? Eg get pgp up and stable, then also allow posting authenticated by IP address. This would mean you could implement something that works and when you come up with something better, you could implement it for no loss except the time involved.

Quote:

quote:
As you said in another message- this is complex because of line breaking. We also need to deal with outlooks love of the "indented" reply and filtering that as appropriate. For that matter, we need to figure out how to filter all the HTML garbage that Outlook (and others) add to the e-mail.
The HTML problem is easy. Block multipart messages. Failing that block multipart emails that don't have a plain text component. I'm biased because I don't like HTML emails, but still this would be an easy fix. Also if emails only go out in plain text you are more likely to get plain text back.

regards
David Cameron

KenSchaefer June 20th, 2003 03:50 AM

WRT to point 1 and IP address suggestions:

I think the IP address thing could be a problem.
The only IP address you can definately authenticate is that of the SMTP server that is sending the mail to your (ie Wrox's) SMTP server. That will probably be the IP address of the ISP's mailserver.

Every other IP address can (and, with spam, is very often) spoofed. Dialup users will also have multiple IP addresses (and potentially a lot of them if they belong to a large ISP that has many b-Class or C-class subnets).

One possibly alternative would be for the moderators to only allow certain posters to "reply via email" (ie people you trust). They could be given a pass-phrase to include at the top of each message (it could be included automatically when the original message goes out to the "trusted poster"). When I reply, this pass-phrase is automatically included in the reply, and checked for by the Wrox email system (this is kinda similar to how LSoft's Listserv allows administrators to change list configuration/moderate via email).


Cheers
Ken

www.adOpenStatic.com

bluckcuck June 20th, 2003 08:44 AM

Quote:

quote:Originally posted by KenSchaefer
 They could be given a pass-phrase to include at the top of each message (it could be included automatically when the original message goes out to the "trusted poster"). When I reply, this pass-phrase is automatically included in the reply, and checked for by the Wrox email system (this is kinda similar to how LSoft's Listserv allows administrators to change list configuration/moderate via email).
I think Ken's idea is interesting and expanding on it may also solve another technical problem .

The problem revolves around matching the reply email back to the correct topic in the forums. In the Snitz database structure there are two tables that house postings: TOPICS and REPLY. The TOPICS table links the posting to a category and forum and contains the original topic message. The REPLY record links to the TOPIC record. This structure allows topics to be moved from one forum to another and have all the replies remain attached. This is also why when a topic is deleted, all its related replies are removed through a delete-cascade mechanism.

When the email reply comes back, we need to determine some classifying information to correctly update the database.

First: Which forum does this reply belong to? This one's relatively easy to resolve as the receiving mailbox would be associated with a forum.

Secondly: Which topic is this a reply to? This is more complicated as simply parsing the subject after the "re:" and looking for a topic match is not guaranteed to be reliable. There's no requirement in email replies that standardizes the reply subject line. Also, there is a possibility of more than one topic in a forum having the same subject.

As a possible solution, I propose expanding on Ken's idea to have the passphrase be effectively a "one-time use" posting password. This posting password would be encoded with a topic ID number, and information positively identifying the forum member that is allowed to use the password (maybe by encoding their password or some other private key contained on the MEMBERS record in the forum).

The downside to this might come from out of office replies, delivery notifications, etc.

One advantage to having to include posting password or PGP signature is that it ensures that the person replying is "really there" and solves points 1,4,5 & 6. By actively needing to add the password, automated systems that reply will be automatically blocked. By using the passphrase method, automated systems would not necessarily be blocked as they could reply by quoting the original message - which would include the passphrase. I have seen delivery notifications, out of office, and delivery warnings that work this way.

Bruce Luckcuck
Director, Applications & Support Services
Wiley Publishing, Inc.

Daniel Walker June 23rd, 2003 05:36 PM

Well, the argument for using a listserver is a strong one. Listservers do not work the way they do because the people who make them are lazy and unimaginative :). In many cases they are built on nearly three decades development work. As such they form the very best solutions to the problem of running a mass mailing system that exists, and are amongst the most highly evolved of all networking daemons. It would be hard to improve upon the vast legacy of high quality code to be found in the average listserver, and Lyis is far from an average listserver... To paraphrase that learned cove from Warwickshire: "A wheel, by any other design, is just as round".

You could, not so much BLOCK, as invalidate, the more offensive spam by simply disallowing HTML mail (only those Chinese agricultural merchants in Guandong and the people who want to sell me a septic tank ever bother to send me any PLAIN TEXT spam). It would sort out a good deal of your Web interface problems at the same time, if you blocked HTML mail, I might add!

You can't stop Out of Office replies since, as I believe Ken Schaefer has already said, the reply goes straight from the recipient to the sender: your mail engine is never involved. Short of insisting that none of your subscribers ever take holidays, you _could_ switch on the "precedence bulk" header, which will often stop this in some cases, but it will also cause many mail filtering systems to block all messages from the listserver on the basis of being suspected spam. That's the way it goes, I'm afraid, though.

As for _which_ listserver to use... Wiley should have inherited the Lyris Platinum lisence Wrox bought to run the P2P lists upon (not to mention the twin-processor Dell Poweredge with the half a Gig of RAM and the SCSI RAID array, that they were running off, and the Dell T550 Windows Advanced Server machine that fed off it - plus the ASP interface that Dave Long built for it).

Now, that Poweredge could send about four times as much mail as the cable running into Arden House could carry away - and it was managing the subscriptions of about 40 000 real individual subcribers towards the end! It also sent the Developer's Journal out every month to another 250 000 subscribers. It was a very powerful machine, and even if you no longer have it, you could replicate it very easily and get the software to run it for the cost of a phone call :). As for customising the interface... well, it was running on very powerful and immensely customisable SQL-base Postgre database on Debian. PostgreSQL supports transactions, SPROCs, and whathaveyou - and there are the Windows ODBC drivers for ASP (or you could even use a PHP interface running on another Linux box).

Certainly I agree with your general point: the less money it costs the better. Don't let people just start chucking money around in a general panic: I've seen how that story ends ;).

Anyway, take it easy, and best of luck with it,
Dan

Quote:

quote:Originally posted by KenSchaefer
 Hi Hal,

I wasn't trying to say that we should use Lyris or LSoft. I'm saying that these products already do most of what you need, so, we should look at how *they* handle these issues and replicate them, rather than rebuilding the wheel. I'm pretty sure you can still get evaluation editions of both of those two products.

Cheers
Ken

www.adOpenStatic.com

Hal Levy June 23rd, 2003 08:16 PM

Quote:

quote:Originally posted by Daniel Walker
 Well, the argument for using a listserver is a strong one.
Not going to happen. That's the final answer. Let's stop talking about what if and use what we have the best we can.

Quote:

quote:
You could, not so much BLOCK, as invalidate, the more offensive spam by simply disallowing HTML mail (only those Chinese agricultural merchants in Guandong and the people who want to sell me a septic tank ever bother to send me any PLAIN TEXT spam). It would sort out a good deal of your Web interface problems at the same time, if you blocked HTML mail, I might add!
Since the mail FROM the server will soon be HTML formatted- and by default outlook sends messages in HTML format- this isn't really workable, unfortunately.


Quote:

quote:
You can't stop Out of Office replies since, as I believe Ken Schaefer has already said, the reply goes straight from the recipient to the sender:
This is usually the case, yes- however some servers send to the FROM: which would be the list- not the person sending it.

Quote:

quote:
As for _which_ listserver to use... Wiley should have inherited the Lyris Platinum lisence Wrox bought to run the P2P lists upon (not to mention the twin-processor Dell Poweredge with the half a Gig of RAM and the SCSI RAID array, that they were running off, and the Dell T550 Windows Advanced Server machine that fed off it - plus the ASP interface that Dave Long built for it).
If this was only true... The Lyris server was at a co-location facility and the machine was leased. The leasing company repossessed the machine WHILE the Wiley employees were trying to download the archives from the server, this is why we are missing the last two months of posts- the data was corrupt on the download and they recovered as much as they could. The version of Lyris that was being used was 2 versions old. To upgrade to a current version was expensive (Wiley uses Lyris elsewhere in their organization- specifically to manage the dummies mailing lists).

The source code for the web interface was completely lost- since there was no employees at "classic wrox" no one could provide it- and it was not downloaded off the co-located server before it was repossessed. I am told the server was actually POWERED DOWN by the co-location facility while Wiley was connected to try and get the files and source code.

They used what they had to try and get the site back up- however it was completely unstable since they didn't get everything. (You might remember the lists went back up, as was, for a day or so) The decision was made to scrap the old system- they had to start over.

Wrox had done some heavy modification to the Lyris system - not even considering the front end. There was no source code for this as it is/was in the hands of classic wrox employees that are unknown.

Wiley was faced with a decision- How do we get P2P up QUICKLY- It had been offline already for too long! Well, Lyris was going to cost a lot of money, require lots of resources, and take a long time to put up. Snitz they felt they could get online quickly and met their requirements for the web interface. At the time they only had web traffic information- and they knew it was considerable.

So, if what you said was true, that they could have just picked up the server and plugged it in somewhere else, I'd agree with you- They did the wrong thing. However, reality is quite different.

Quote:

quote:
It was a very powerful machine, and even if you no longer have it, you could replicate it very easily and get the software to run it for the cost of a phone call :).
Not quite- the lyris software was due for an expensive upgrade and the machine was long gone. The lyris system, as I said above, was customized by Wrox people. Customizations that wiley didn't have any source code for.

Quote:

quote:
As for customising the interface... well, it was running on very powerful and immensely customisable SQL-base Postgre database on Debian. PostgreSQL supports transactions, SPROCs, and whathaveyou - and there are the Windows ODBC drivers for ASP (or you could even use a PHP interface running on another Linux box).
Ah- another issue- They have nothing but windows servers at Wiley. They have no Linux experience. They have no PostgreSQL experience.

So, in short, they had no server, no source code, an out of date Lyris licence, a corrupt database, a front end that (even according to classic wrox employees) needed daily reboots to stay online and no experience with Linux.


I agree, I liked using e-mail. I thought it was better. Certainly more convenient. Faster to load. I could take it with me on my laptop. I could sort, organize, group, fiddle with, and muck around how ever I was comfortable using it. However, from my extended conversation with the "powers that be" it's been made clear to me that if there was going to be a P2P at all, this is how it was getting done.

SO here we are- with Snitz and there's no way that's changing. So, Dan, are you going to help us make it more mailing list like?


Hal Levy
NOT a Wiley/Wrox Employee- Got a job for me?

Daniel Walker June 24th, 2003 05:19 AM

[quote]quote:Originally posted by Hal Levy
 
Quote:

quote:Originally posted by Daniel Walker
Quote:

 Well, the argument for using a listserver is a strong one.
Quote:

quote:
You say:
Not going to happen. That's the final answer. Let's stop talking about what if and use what we have the best we can.

Then you say:
Since the mail FROM the server will soon be HTML formatted- and by default outlook sends messages in HTML format- this isn't really workable, unfortunately.
If there's mail coming [i]from[/] a server, then it's coming from a mail listserver. So, you can:
a) write one yourself
b) use one you buy
c) use the one we bought
Quote:

Then I comment:
As for _which_ listserver to use... Wiley should have inherited the Lyris Platinum lisence Wrox bought to run the P2P lists upon (not to mention the twin-processor Dell Poweredge with the half a Gig of RAM and the SCSI RAID array, that they were running off, and the Dell T550 Windows Advanced Server machine that fed off it - plus the ASP interface that Dave Long built for it).

You reply:
If this was only true... The Lyris server was at a co-location facility and the machine was leased.
Well, unless it was moved in the last days at Wrox (quite possible, I suppose, although Stephen Biggerstaff would be the man to ask about that), I know that (as an ex-"classic" Wrox employee) the listerver machine was a big black Dell Poweredge 4100 that sat sideways-on at the end of the shelf in the server room inside Arden house. The webserver was a Dell Dimension T550 that sat beside it running Windows 2000 Advanced Server. Both machines and the software they ran were wholely owned by Wrox - at least at some time in or around Jan/Feb.

There was talk of moving it to Chicago, but I don't know if that was done. Either way, I can't imagine the Listserver or webserver were pressed back into service - they should have been in there somewhere.

Stephen Biggerstaff would know what exactly happened to the listserver. Dave Long bult the web interface. I can probably provide a contact address, if I get in contact with him first, to ask if he's interested in helping. Furthermore, there was a blue folder with documentation in it, with printouts of the ASP source code.

I know a decision seems to have been made to not use this option, but it seems a shame to dismantle what was running and maintaining 40 000 active subscriptions, for want of not knowing which machines have the source code on them. The Lyris licence might not have been the latest version (you have Wiley talking to you directly about that, so I can only accept that this was so), but I know a Platinum licence was bought for it last Autumn - a licence for the SQL-based modern Lyris, which Wrox did pay for. What's more, if classic Wrox employees told you the P2P webserver needed rebooting every day, they were mistaken. Anyone who was using this site in the last days will tell you that the lists continued running for at least a week after the closure - as Mike Kay and others commented at the time.

Wiley are the new owners, and they made much of how hard it was to tell what was what when they took over Wrox: it was certainly a mess - for understandable reasons, I hope - and I'm certainly not going to tell them how to run their new acquisition, but there are ex-Wrox, ex-glasshaus, ex-FoED employees who may well try to help, if all it takes is to identify which machines were which and where to find stuff. It's probably too late, now, though.

Dan

David Long June 24th, 2003 06:22 AM

Quote:

quote:Originally posted by Hal Levy
The Lyris server was at a co-location facility and the machine was leased.
The Lyris machine was most definitely located at Wrox's offices in Arden House, up to and for at least a couple of weeks after the closure of Wrox. I visited the building over a week after the redundancies and the machine was still there.

As for the leasing issue, as far as I was aware that particular machine was wholly owned by Wrox (unlike a number of desktop machines in the same building), though there was an issue with unpaid rent for Arden House and the remaining machines may have been sent to auction to pay off this debt.

Quote:

quote:Originally posted by Hal Levy
The source code for the web interface was completely lost- since there was no employees at "classic wrox" no one could provide it- and it was not downloaded off the co-located server before it was repossessed.
Where did http://p2p.wrox.com/archive/index.php/ come from, then? Wiley seem to have an intact copy of that. In fact, I can prove that some of the code for the web interface still exists; visit Google and search for "p2p.wrox.com xslt archive". The first hit will be "p2p.wrox.com/archive/xslt/2001-09/76.asp", if you click through from Google to that page you will receive an error at line 114 in /include/header.asp. This is because of a referrer check that I coded to try and encourage people who found the archive via Google to subscribe; it adds a couple of extra lines to the top of each page. If Wiley have the whole of /archive and /include/header.asp (and also, I guess, /include/footer.asp and a few related other files), why don't they have the rest of the web interface? The bulk of the work was done by only a handful of files located in the web root.

Incidentally, someone at Wiley might want to dig into /include/header.asp and fix that bug. From what I remember of the old web stats, at least 25% of the hits we received in the archive were referred from Google, and that error won't be giving people a good impression ;)

Quote:

quote:Originally posted by Hal Levy
Wrox had done some heavy modification to the Lyris system - not even considering the front end. There was no source code for this as it is/was in the hands of classic wrox employees that are unknown.
We didn't make any "heavy modification" to Lyris itself; it was a pretty much bog-standard install of Lyris/PostgreSQL running on Debian 3.0. The only non-standard extra we installed was a Perl script that stripped HTML from incoming email (incidentally, you might want to use a similar script in the new email system?)

As for the "unknown wrox employees", they are myself, Dan Walker and Stephen Biggerstaff. We're all registered on this new forum, and I at least offered my services to Wiley regarding P2P by private email a while back. I heard nothing whatsoever from them, though.

Quote:

quote:Originally posted by Hal Levy
Wiley was faced with a decision- How do we get P2P up QUICKLY- It had been offline already for too long! Well, Lyris was going to cost a lot of money, require lots of resources, and take a long time to put up.
Wiley could have spoken to me, I'd have been more than willing to help. I really didn't want to see P2P disappear off the face of the 'net, seeing as I'd worked on it for over two years, but nobody bothered to get in touch with me.

Quote:

quote:Originally posted by Hal Levy
SO here we are- with Snitz and there's no way that's changing.
Fair enough, that's Wiley's business decision to make. I'm still convinced it's the wrong decision, and that you won't attract the number of active, consistently good posters that we used to see in the old days of P2P. Having said that, I'm prepared to be proven wrong, though :)

StephenB June 24th, 2003 06:58 AM

Hello Dan and Dave, how are you? (stephenb@orange.net for any off topic converse)

I in fact left Wrox in January but continued administering P2P on a freelance basis until the liquidation. Dan and Dave are correct, most of what Hal says seems to be inaccurate second-hand information, sorry Hal. We don't know exactly what happened to the servers from the liquidation until Wiley agreed to take over, although P2P was still functioning for news and mail access for quite a while afterwards, even when the webserver was down, one of the things which made it so robust. (No, it didn't need rebooting every day.)

As soon as I knew Wiley were taking over, I contacted Joe Wickert and offered to help, also mentioning Dave's name as the main developer. I think Joe passed our names to the Wiley tech guys, but we were never asked for our help or advice.

While I'm here I'd like to clear up one or two other misconceptions...

Since approximately November 2001, P2P has been effectively unmoderated. I only had time allowed for support issues and list monitoring, banning the occasional miscreant. One effect of this change was to increase the amount of traffic as there was no delay in posting to the list!

Despite this spam was never a problem, though Wiley are right to worry that it might become one, the amount of spam was increasing slowly. Similarly email address harvesting was becoming more noticable, but still at a low level.

I think quite a few people at Wrox didn't really understand what P2P was all about, or realise that it was more of a news and mail community than a web forum community. (One of the reasons that Wrox eventually went under...?) That misconception seems to have been passed on to Wiley. It is understandable though that if the guys at Wiley are happier with web forums and not happy with a Linux based system that they weren't interested in taking the machines, even if they had the chance.

As Dave says, I hope P2P does well with Wiley, but as many of the knowledgable people that answered a lot of questions used news or mail for the convenience, it will be an uphill struggle. The number of users and amount of traffic are still much much lower than previously.

Stephen

Hal Levy June 24th, 2003 07:12 AM

[quote]quote:Originally posted by Daniel Walker
If there's mail coming [i]from[/] a server, then it's coming from a mail listserver. So, you can:
a) write one yourself
b) use one you buy
c) use the one we bought
[quote]quote:

Dan, as you know, Snitz sends e-mail already built into the package. It does not have a method of receiving e-mail and turning it into a post on the forum. So it isn't a "listserver" in the true sense of the word. It's, in my opinion, a web based forum with e-mail notification.


Quote:

quote:
Well, unless it was moved in the last days at Wrox (quite possible, I suppose, although Stephen Biggerstaff would be the man to ask about that), I know that (as an ex-"classic" Wrox employee) the listerver machine was a big black Dell Poweredge 4100 that sat sideways-on at the end of the shelf in the server room inside Arden house. The webserver was a Dell Dimension T550 that sat beside it running Windows 2000 Advanced Server. Both machines and the software they ran were wholely owned by Wrox - at least at some time in or around Jan/Feb.
This is different than I was told by the people at Wiley. The technical people (James and Bruce) told me the information that I have passed on here. So the question becomes - Did the liquidators not give them everything they paid for? Did they sell the hardware out from under Wiley? Or did Wiley leave some parts out of the discussion I had with them?

Quote:

quote:
Stephen Biggerstaff would know what exactly happened to the listserver. Dave Long bult the web interface. I can probably provide a contact address, if I get in contact with him first, to ask if he's interested in helping. Furthermore, there was a blue folder with documentation in it, with printouts of the ASP source code.
Now, if it was up to me (and it's not, I don't even get to sit in on their meetings) if all the source code and the archives and all that are really available then I would be looking into switching back.

Personally, in that case, I would propose that this forum stay up in it's current form- and the web front end (which, I am told, needed reboots daily) be re-developed and in this way the front end would be built upon Lyris. Personally I think it much easier to build a web front end on the server than the other way around.

However, it's not up to me...

[quote]
I know a decision seems to have been made to not use this option, but it seems a shame to dismantle what was running and maintaining 40 000 active subscriptions, for want of not knowing which machines have the source code on them. The Lyris licence might not have been the latest version (you have Wiley talking to you directly about that, so I can only accept that this was so), but I know a Platinum licence was bought for it last Autumn - a licence for the SQL-based modern Lyris, which Wrox did pay for.
{/quote]

It seems like there's information that (at this time) all I can say is unknown to the guys at Wiley. I would hope if they had all the information (and got all of it from the liquidators) that they were told- however they might not have been.


Quote:

What's more, if classic Wrox employees told you the P2P webserver needed rebooting every day, they were mistaken. Anyone who was using this site in the last days will tell you that the lists continued running for at least a week after the closure - as Mike Kay and others commented at the time.
Yes, I do know it ran for a while after the liquidation was announced- however, SOMEONE rebooted it a few times in that period, because there were outages during the weeks following the announcement and the web site returned. A simple AT job could reboot the webserver nightly, however, with no employees needed.

Quote:

there are ex-Wrox, ex-glasshaus, ex-FoED employees who may well try to help, if all it takes is to identify which machines were which and where to find stuff. It's probably too late, now, though.
Personally, I don't know that they got any machine to work with. I know James and Bruce were working remotely on the machines- I think wiley didn't get things like physical machines.... Only the contents and then they had to grab them remotely. However, I will leave that to them to comment on- as I don't really know... If your information is correct, it seems like quite a bit of value was lost when the transition took place. Value that didn't have to be lost.


Hal Levy
NOT a Wiley/Wrox Employee- Got a job for me?

Hal Levy June 24th, 2003 07:39 AM

Quote:

quote:Originally posted by David Long
Where did http://p2p.wrox.com/archive/ come from, then? Wiley seem to have an intact copy of that. In fact, I can prove that some of the code for the web interface still exists; visit Google and search for "p2p.wrox.com xslt archive". The first hit will be "p2p.wrox.com/archive/xslt/2001-09/76.asp", if you click through from Google to that page you will receive an error at line 114 in /include/header.asp. This is because of a referrer check that I coded to try and encourage people who found the archive via Google to subscribe; it adds a couple of extra lines to the top of each page. If Wiley have the whole of /archive and /include/header.asp (and also, I guess, /include/footer.asp and a few related other files), why don't they have the rest of the web interface? The bulk of the work was done by only a handful of files located in the web root.
David, I can't answer that. Hopefully some of the Wiley people can. I have noticed that parts of the web interface seem to have survived in some places. I am told, however I don't know for sure, that most of the last two months of messages are missing from the archive. I do know none of my April posts on design_patterns are in the archive.

Quote:

quote:
We didn't make any "heavy modification" to Lyris itself; it was a pretty much bog-standard install of Lyris/PostgreSQL running on Debian 3.0. The only non-standard extra we installed was a Perl script that stripped HTML from incoming email (incidentally, you might want to use a similar script in the new email system?)
Interesting to know David. I personally at this point feel a bit out on a limb-- as I was told all these items by the wiley staff and it seems as if there's some information that was either not told to them or that they were unwilling to tell me. and yes, I plan on having to use something similar in the new system.

I even asked them dead-on about why the Lyris system (and web site) came back online for a short time. They told me they got some of it and tried to get it running and they couldn't keep it online and that the web interface was crashing.

Quote:

quote:
As for the "unknown wrox employees", they are myself, Dan Walker and Stephen
Biggerstaff. We're all registered on this new forum, and I at least offered my services to Wiley regarding P2P by private email a while back. I heard nothing whatsoever from them, though.
Um, I know I have not spoken to you or Stephen or Dan directly about the P2P stuff and the machines and reboots. I spoke to Jan about the webserver at one point and he told me that he thought it was rebooted daily because there were stability issues. However, at this point, I am not so confident I am remembering everything correctly.

If this is incorrect, I certainly don't want to be a person on record insulting your work as being unstable when it wasn't!

Quote:

quote:
Wiley could have spoken to me, I'd have been more than willing to help. I really didn't want to see P2P disappear off the face of the 'net, seeing as I'd worked on it for over two years, but nobody bothered to get in touch with me.
I can't speak for them- Perhaps there was a reason they couldn't. You say you emailed them, so I expect they knew who you were- I really don't know. But the more I hear from you, Dan and Stephen the more I want to hear more from Wiley....


Quote:

quote:Fair enough, that's Wiley's business decision to make. I'm still convinced it's the wrong decision, and that you won't attract the number of active, consistently good posters that we used to see in the old days of P2P. Having said that, I'm prepared to be proven wrong, though :)
This is agree with 200%. I do NOT like that they moved to a web forum. I can tolerate it- and I have been- but I liked e-mail SO much better. The involvement was better from both experts and the authors.


Is it true that p2p was actively moderated to keep spam off the list? And if so- how did it run until 5/2 (the last time I got a message from the server) without that moderation- since there were no employees.... More questions than answers it seems.

I'm thinking it's time for me to stop defending wiley and let the employees do that, if they want. It's not my company, they don't even pay me a salary (and I could use one!) The fact is, I am interested in getting help for my questions and answer those I can, and if this is that forum- so be it. Perhaps APRESS will come out with a Developers forum using a true mail listserv.


Hal Levy
NOT a Wiley/Wrox Employee- Got a job for me?

Hal Levy June 24th, 2003 07:50 AM

Quote:

quote:Originally posted by StephenB
I in fact left Wrox in January but continued administering P2P on a freelance basis until the liquidation. Dan and Dave are correct, most of what Hal says seems to be inaccurate second-hand information, sorry Hal.
I freely admit I have information that's second-hand. I have been told things by the tech guys at Wiley- I assume their information is correct and look forward to them responding to the questions you, David and Dan have raised in this thread- since I can't. I retract everything that you guys say isn't accurate- since you were there and I wasn't.

Quote:

quote:
As soon as I knew Wiley were taking over, I contacted Joe Wickert and offered to help, also mentioning Dave's name as the main developer. I think Joe passed our names to the Wiley tech guys, but we were never asked for our help or advice.
This raises a huge problem for me as I was given the impression they had no idea who from classic wrox could assist them.

Quote:

quote:
Since approximately November 2001, P2P has been effectively unmoderated. I only had time allowed for support issues and list monitoring, banning the occasional miscreant. One effect of this change was to increase the amount of traffic as there was no delay in posting to the list!
See, that's what I said, I said that there was too much traffic and my posts went up too fast. The impression I was given was there was a moderation team- obviously this is wrong.

Quote:

quote:
I think quite a few people at Wrox didn't really understand what P2P was all about, or realise that it was more of a news and mail community than a web forum community. (One of the reasons that Wrox eventually went under...?) That misconception seems to have been passed on to Wiley. It is understandable though that if the guys at Wiley are happier with web forums and not happy with a Linux based system that they weren't interested in taking the machines, even if they had the chance.
The only volume numbers they had were from the website- which they felt was considerable. (or so they told me) Personally I think what made Wrox go under was the complete loss of focus on what made them great in the first place. A series of clearly defined books that took you from novice to Expert in 3 or 4 books. In the end, Wrox was pushing out books that had so much overlap, we had no idea what to buy- so I just stopped buying books from Wrox.. I was buying more and more APRESS and MSPress books over the last year because I was just confused by what WROX was putting out there.

Quote:

quote:
As Dave says, I hope P2P does well with Wiley, but as many of the knowledgable people that answered a lot of questions used news or mail for the convenience, it will be an uphill struggle. The number of users and amount of traffic are still much much lower than previously.
Stephen, unfortunately I think too many knowledgeable people were lost. On the old forums my postings were barely a speck. I certainly wasn't answering as many questions as I do now- of course, that was because I didn't need to, so many others were out there answering them.

Hal Levy
NOT a Wiley/Wrox Employee- Got a job for me?

Daniel Walker June 24th, 2003 08:10 AM

You will forgive any tone of irate outrage that may appear to come through from us: we just want to help and we want to set the record straight. There has been much made by Wiley of the "state of chaos" they discovered at Arden House - especially with ex-authors - and this does not reflect well upon a very hard-working and dedicated workforce, as I'm sure you will realise.

The truth,as Philip K Dick once observerd, is "that which does not go away when you stop believing in it". There are several things I have stopped believing in since my time at Wrox ended, but the truth still remains the same and I recognise no gain in seeing it abused - even if only through misunderstanding.

Personally, I hope one day that I will work again in an environment surrounded by such a high density of gifted, clever, good natured, honest, and hard working people as I found a Wrox.

Lastly, (and this is merely a tongue-in-cheek observation and is not intended as a parthian shot, I hope you understand) it is perhaps unwise to advertise the fact that a computer programming publisher has "nothing but windows servers", has "no Linux experience", "no PostgreSQL experience", et cetera :).

Dan
(who regards the phrase "Classic Wrox" in much the same light as teh phrase "Classic Mac OS", and regards both as something best remembered with affection, rather than actually experienced :).

David Long June 24th, 2003 08:31 AM

Quote:

quote:Originally posted by Hal Levy
I have noticed that parts of the web interface seem to have survived in some places. I am told, however I don't know for sure, that most of the last two months of messages are missing from the archive. I do know none of my April posts on design_patterns are in the archive.
The archives were created by a separate process running on the webserver, which wasn't as stable as we'd have liked (though "daily reboots" are far from the truth). This process probably failed some time towards the end of Wrox's existence, which is probably why those posts don't show up.

Quote:

quote:Originally posted by Hal Levy
I even asked them dead-on about why the Lyris system (and web site) came back online for a short time. They told me they got some of it and tried to get it running and they couldn't keep it online and that the web interface was crashing.
The web interface was pretty much hard-coded to work on Wrox's network only, relying on fixed IP addresses, system DSNs and suchlike for it to work correctly. I'm not particularly surprised that Wiley couldn't get it back up quickly after transferring it to their servers, but with a bit of work (and help from me, if they needed it) I'm sure it could've been put back into service for a short while at least until a newer interface was ready. Some documentation was also available on paper, but I doubt that Wiley had access to that.

Quote:

quote:Originally posted by Hal Levy
Um, I know I have not spoken to you or Stephen or Dan directly about the P2P stuff and the machines and reboots. I spoke to Jan about the webserver at one point and he told me that he thought it was rebooted daily because there were stability issues. However, at this point, I am not so confident I am remembering everything correctly.

If this is incorrect, I certainly don't want to be a person on record insulting your work as being unstable when it wasn't!
I wouldn't say that the P2P web interface was the most stable thing I've ever written; it was the first large-scale ASP site I ever worked on, and as such the code wasn't particularly clean, but it did work - two years and over 20 million page views (if I remember correctly) aren't to be sniffed at.

This "daily reboots" thing is indeed just a myth; IIS might've needed a restart every few weeks, but as far as I know that's pretty much necessary on most IIS installations every now and again. Not to be insulting to Jan (I worked with/for him for a significant amount of time at Wrox), but sometimes he exaggerated things just a bit :)

Quote:

quote:Originally posted by Hal Levy
Perhaps APRESS will come out with a Developers forum using a true mail listserv.
I'm wondering whether, between us, we could do this on our own, if Wiley don't want to implement such a thing... hosting/bandwidth costs are the only real problem here, using free software would take care of most of the actual work involved. Pete Aylward from Friends of ED kept the FoED forums running after the collapse of Wrox, and now Apress seem to be working with him in carrying them on in the same format - see http://friendsofed.infopop.net/

Dave.

David Long June 24th, 2003 08:43 AM

Quote:

quote:Originally posted by David Long
The archives were created by a separate process running on the webserver, which wasn't as stable as we'd have liked (though "daily reboots" are far from the truth). This process probably failed some time towards the end of Wrox's existence, which is probably why those posts don't show up.
In fact, looking at the archives, it seems that posts up to March 26 were archived correctly. Seeing as there was nobody at Arden House to administer the machines from close of business on March 14 (the day of the announced bankruptcy) onwards, that's pretty good going for a system that needed "daily reboots" ;) The Lyris server survived even longer than that from what I know, but unfortunately I think all those messages have been lost now...

JSample June 24th, 2003 09:44 AM

Hello all.

I want to take a few minutes to respond to some of your questions and concerns but first I want to say that this will be the last time that I do so. These questions have completely taken over the focus of Hal’s post which is a request for help in adding a feature that so many of you say that you want.

1. I can answer most of your questions simply by saying that we bought the Wrox assets from a liquidator that had no technical knowledge of what they were selling us. What we got was incomplete, corrupt, and came with little or no explanation.
2. Yes we do have all of the source code for the p2p website. What we said was the dump from the postgress database that we received was corrupt and we were unable to get a new dump.
3. The only thing we new about the web interface was that from the time we started work on the conversion (In May) an IIS reset we required each morning on the server(s) in the UK. IIS reset is something a site should never need, our primary web farm currently hosts 17 websites and I have not had to reboot the servers or do an IIS reset in 8 months.
4. Whether you disagree with our decision or not we have stated many time why we made the decision to switch to the web forum.
5. As for Help in getting the old site up and running, we did hear that someone would be willing to help. But since we were told the servers were leased (which we were told on several occasions) the skill sets required for a new implementation of the site on our standard systems would be very different than the skills needs on the old system. But the time to re-develop the backend database, front-end website, and other features was just too long and expensive.

What this all means is that for all of the information flowing back and forth, there is still ONE feature missing from making this site the best of both worlds. I would ask again that if the feature of replying is truly wanted than please lend Hal your help in solving the 7 issues and we will be happy to implement the system adding back the ability you are lacking.

Thank you,


James Sample
Director, IT-Infrastructure
Wiley Publishing, Inc.

David Long June 24th, 2003 10:21 AM

Please accept my apologies for taking this thread off-topic, but I, Dan and Stephen thought it best to try and clear up things that had already been mis-reported.

Back on topic: Many suggestions have already been given in this thread, but some of the problems seem unsolvable:
  • Authenticating users by something other than the "from" address - not difficult when dealing with replies (the authentication code would just be included in the reply) but new threads started by email would need something extra included in the body, which people just won't remember. We tried reminding people on the old site to cut down on the amount they quoted and to not quote headers/footers, but it just doesn't work, people just want to type their message, send it and forget about it - they don't want to have to take extra steps just to post to a mailing list, especially if they don't have to do that on other mailing lists they use.

    The "registered IP address" mechanism probably won't work reliably in most cases due to SMTP relays (this will affect most users to some extent, I'd have thought), and insisting on PGP signatures will alienate users who currently don't use them. I can't see an alternative, workable solution for this, and nobody else has posted one in this thread either.
  • Detection of overquoting is never going to be foolproof. I tried many times to write some code that could handle any quoting mechanism so the archive would be a bit neater, but I gave up in the end as I figured it's just not possible - there's simply too many mailers out there that do it in different and incompatible ways.

    If you really want to achieve this, I can't see how you'll do it without a human operator in the loop to figure out exactly which bits of the original message are relevant, and which bits aren't. A computer program would basically need full understanding of the English language in order to be able to do this successfully, even if you can figure out which bits of a message are quotes and which bits are new text - say you know you've got 1000 lines of quoting and 10 lines of new reply in a particular message, which of the 1000 lines do you keep, if you can't understand the context?
  • Figuring out replies to topics and attaching them to the correct threads is technically possibly by subject line (this is how the old web interface list browser and archive worked - strip "Re:" and similar prefixes and all whitespace to give you a unique topic string), though it's not 100% foolproof. It did work well enough for us, most people use unique enough subject lines to be able to figure it out.

    Alternatively, you could include a unique identifier at the start of the subject line, which would be intact in the reply. You could even use this to implement true threading rather than all replies being attached to a single root-level topic, but as far as I can see Snitz doesn't support this anyway.
  • Out of office and other automated replies can mostly be blocked by scanning incoming messages for known substrings, but again this won't be 100% perfect. There'll always be someone with an out-of-office reply in a foreign language that doesn't get trapped with this method.
  • Spam filtering can be achieved by placing SpamAssassin, a Bayesian filter or some similar software between the receiving mail host and the software that puts the message up on the forum. Again, this isn't going to be 100% perfect; determined spammers will always find a way around automated checks eventually (if they couldn't, spam wouldn't be a problem for anyone!). But, like we've already said, spam wasn't a problem on the old forums, and with a few extra checks in place, I can't see it being a problem on the new ones either.

As has already been said, the easiest and cheapest way to integrate this would be to use a mailing list manager that someone else has already written, then try and write an interface to bolt it into the Snitz forums, but if Wiley don't want to use a third-party product, then I'm stumped as to how to proceed further with this, especially the authentication requirement.

Finally, credit must be given to Hal for talking to Wiley and trying to make the forum usable for the masses by taking it back to the ways that everyone was familiar with. I don't think I'd have taken on that job :)

Cheers,

Dave.

JSample June 24th, 2003 10:35 AM

I would not say that we will not consider a 3rd party product. We already use Lyris List manager here in Indy so we would definitely consider using it. What we said we wouldn't consider was dumping the web forums that are there for a starting over in developing a web front end to Lyris. For good or bad we have already started down one path and don't want to backtrack.

As far as some things not being fool proof I can say that nothing ever is. All that can be done is our best and we can see how good a job it does and if it is good enough to implement anyways. The contraints cannot be ignored but I think the level at which they are addressed is certainly up for some debate.

Thank you,

James Sample
Director, IT-Infrastructure
Wiley Publishing, Inc.

David Long June 24th, 2003 10:44 AM

OK, so we can basically ignore all the constraints except the authentication one for now, as I think we've solved those to at least some extent in this thread. Does anyone here see how the authentication requirement can work without being too intrusive on the users?

From the suggestions posted so far, IP address checking just won't be reliable enough, so that leaves you with either PGP (most people don't use this, and good luck in trying to convince them to just so they can join a mailing list) or passwords/unique codes in the message body (most people will forget to add these after their first posting). Are there any other workable alternatives?

This is really the only sticking point with the list of requirements. If there's no feasible technical solution to this authentication problem, does that mean we won't be getting an email interface to this forum? Have the developers at Wiley seen this thread? I'd be interested to get their input on our remarks, especially regarding how they see the authentication process (possibly) working...

Cheers,

Dave.

[edited because I hit send too early, then again because I added an extra sentence]

Daniel Walker June 24th, 2003 11:39 AM

PGP is also a fairly processor-heavy operation. It's fine when only two machines are inviolved, at either end, but as soon as a server in between becomes involved, handling lots of messages at once, it could tie itself in knots, couldn't it?

Or has someone already made that point? (blushes and sidles out)

Dan

Quote:

quote:Originally posted by David Long
 Does anyone here see how the authentication requirement can work without being too intrusive on the users?

From the suggestions posted so far, IP address checking just won't be reliable enough, so that leaves you with either PGP (most people don't use this, and good luck in trying to convince them to just so they can join a mailing list) or passwords/unique codes in the message body (most people will forget to add these after their first posting). Are there any other workable alternatives?

Cheers,

Dave.

Jeff Mason June 24th, 2003 12:05 PM

Does PGP cost anything? Are there licensing issues? Is it available as freeware on all platforms that p2p users might be running on?

If a fully functional version which will run in corporate (e.g. Exchange) environments as well as on the variety of personal systems is not available for free, then many people might not want to participate in a forum system that requires them to go out and buy a piece of software...

Jeff Mason
Custom Apps, Inc.
www.custom-apps.com

chrislepingwell June 24th, 2003 12:11 PM

You have a point here. Aside from the fact that Wiley would have to shell out for the full commercial version, peoples employers' might not take kindly to having 'unofficial' software installed on their desktops, even assuming that the individuals had the ability to do it. Where I work, you need special dispensation even to change the date and time in windows!

There are two secrets to success in this world:
1. Never tell everything you know

David Long June 24th, 2003 12:15 PM

Quote:

quote:Originally posted by Jeff Mason
Does PGP cost anything? Are there licensing issues? Is it available as freeware on all platforms that p2p users might be running on?
There are free versions of PGP for pretty much every platform - see http://www.pgpi.org/products/pgp/versions/freeware/ for details. But having said that, many corporate desktop computers are locked down to such an extent that users can't install any third-party software, and in most cases I guess the system administrators wouldn't install it just because one of their users wants to post to a mailing list. So that (plus the processing power issue) effectively rules out requiring PGP to authenticate postings. We're left with just the "include a password/unique code in the message body" idea, unless anyone has any more bright ideas on how to solve this.

Edit: chrislepingwell beat me to it, sorry if this sounds like a duplicate post :)

Edit 2: the freeware version of PGP more than likely wouldn't be usable by most people anyway, according to the licensing details at http://www.pgp.com/products/freeware.html


All times are GMT -4. The time now is 01:11 PM.

Powered by vBulletin®
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
© 2013 John Wiley & Sons, Inc.