Re: Apache Vulnerability in WebTen + technical support and QA

I have been away from my email for a few weeks, and am just catching up. Although this is an old topic, I must reply.

> >> I too cannot turn on Squid without encountering multiple serious problems,
>> so need to know the extent of my vulnerability until I can move to a new >> server (a slow process, as it is still in budgeting, and as I have a lot of >> learning to do beforehand). > > What are these "multiple serious problems" that you are experiencing with > Squid on?

I had numerous problems and every time I would report them to support,
> Stephanie's answer would be to turn off the cache.
The advice you were given to turn off Squid in order to fix the problems
that you were having is absolutely incorrect.

No, it isn't!

Instead of saying that I was incorrect and dismissing user reports as obviously incompetent, you might instead try listening to what those users have to say about their real-world experience!

 I apologize for the
incorrect advice you were given.

I did not give incorrect advice in this particular case!

Please do not take it upon yourself to apologize for my work!

Although you believe it to be incorrect - I stand by what I have told these users and others!

If any user on this mailing list is unhappy with the tech support I supplied, please feel free to email me directly and tell me you think I'm a jerk (or whatever works for you). My email address is in my signature below. I accept both criticism and compliments. (And supply recipes to all, as well as excess garden vegetables if you're in my neighborhood.)

Erik.

--
Erik Lotspeich                          Lead Engineer
Tenon Intersystems                      erik@xxxxxxxxx
1123 Chapala Street Ste 200             805-963-6983
Santa Barbara, CA 93101-3142            http://www.tenon.com/

I believe that my recommendation to turn off Squid was *not* inappropriate for the 3 WebTen users who chimed in on this particular mailing list discussion. Nor for the many more to whom I made this recommendation via phone or email technical support at Tenon. (I recognize that at this point, with the new security concern in Apache, other solutions may very well need to be found).

Tenon should be commended for being different from some other software companies, in that it does use its own products day-to-day, in a real-life setting. Nevertheless, Tenon's real-life server configurations represent a very small fraction of the many possible configurations that exist planet-wide, among users of their products. Not to mention that the server administrators at Tenon almost certainly have different experience/expertise than the average user of WebTen. (Or iTools for that matter).

I am certain that the QA testing at Tenon, which found wonderful performance of the WebTen + Squid combination was/is completely accurate, at least insofar as the configurations actually tested.

But real people, in the real world, have had quite different experiences, some of them not so positive. That is most likely because they are not using the product in exactly the same manner as it was tested in a laboratory setting.

(What a surprise... that normal mortals might use a software product in a different way... under different circumstances... than the engineers who thought it up... what a concept... wasn't there some Apple ad campaign about thinking different... well guess what, people using software products often do unanticipated things...)

One of the reasons that I have been called an excellent or exemplary technical support person for every product I have supported, is that I *listen* to what users of the software products report about their *actual* experiences.

I work with users to find a solution which works for them, even if that solution is counter to the prevailing dogma.

All three users discussing this on the WebTen mailing list have stated that turning Squid off, in actual fact, did improve both performance and stability for them. Although I'm sure this wasn't what was intended by the product engineer, the response to these user's first-hand observations [of their own servers] implied that they were somehow wrong or inaccurate... this could be interpreted as an insult to these people's intelligence, observational skills, and/or professional competence...

I help people no matter what their level of technical experience - a third-grade school teacher trying get the school's web site up deserves just as much patience and respect as someone hosting a hundred web sites dealing with a very complex problem.

A bonus is that I pay attention and notice patterns when more than one user reports similar experiences.

If one actually listens to WebTen users, you will quickly find out that many of them run other server products on the same machine with a single IP number - Quick DNS Pro, EIMS, SIMS, LetterRip, etc.... frankly, I don't think these server products have anything to do with the poor performance and instability many WebTen users experience when Squid is enabled, but there could be other suboptimal influences...

What I believe is the real factor, something easily discovered if one takes 30 seconds to actually ask WebTeb server users one or two simple questions about their set-up, is.... the vast majority of WebTen users also use one, or more, 3rd plug-ins to the server - the two most popular being Lasso and/or Netcloak. (Web Catalog, Tango and others were also also quite popular at one time).

WebTen(Apache) + Squid do work fine together.

WebTen + 3rd party plug-ins also work fine together.

WebTen + Squid + 3rd party plug-ins = NOT GOOD in many instances.

Even with properly configured DNS and all that, WebTen + Squid + Lasso (or many other 3rd party plug-ins) are not a good combination.

I cannot tell you at a code level why this so, because that is not my area of expertise.

But empirical evidence from dozens of users confirmed my suspicions - Squid does not play nice with many 3rd party plug-ins, particularly those that create dynamic html pages, or those for creating dynamic pages created from database access.

These are the most popular types of 3rd party plug-ins!
Used by 90% or more of WebTen users!

(I can picture some possible ways http requests going through Squid could get messed up since some of the 3rd part plug-ins have built-in caching mechanisms of their own - I can imagine things going into counterproductive loops when more than one caching mechanism is involved in a request for a particular page... but this is purely hypothetical speculation...)

Since many WebTen users have hundreds or thousands of html coded pages with complex Lasso or Netcloak tags (or Web Catalog, Tango, etc) suggesting that they abandon their favorite plug-ins for the theoretical benefits of Squid was not, in my opinion, a reasonable thing for tech support to suggest to these server administrators.

For many of these users, the performance benefit of Squid is moot given their server traffic, hardware or bandwidth, or some other consideration (the size of their FileMaker database or something else which constrains number of pages served per time interval).

My pragmatic solution was to say, "turn Squid off, and see if it helps".

If it didn't, then tech support could move on to other problem solving ideas (rarely needed, since turning off Squid solved most problems).

If turning off Squid did help, *then* there could be evaluation of the possible impact, either negative or positive, on performance. The supposed benefit of Squid is close to nonexistent for many (if not most) real-world WebTen server scenarios (given that close to 100% of WebTen users also use 3rd part plug-ins), therefore a trial period without it is a perfectly reasonable suggestion.

After following the suggestion to try turning Squid off, I was rewarded by many user's reports that a week or so of trying that had resulted in vastly improved performance and better stability. Who cares if Squid is wonderful in the QA lab, if it messes people up in the real world?

Based on all evidence I have to date, I believe that the vast majority of people using WebTen with 3rd party plug-ins (most WebTen users) will have better server response and greater stability with Squid off than on. The three people who commented on this topic on the WebTen mailing list report exactly that from their own experience.

Official response to these user's reports seems to be to state that this is wrong and cannot be so. So are these people deluded by what they have observed on their own servers, or what?

=============================================================================

Most sophisticated software users (people running servers for example), recognize that it is simply not possible, for any QA lab, to test every combination of hardware, server version, 3rd party plug-in(s) or additional applications running on the same network or box, not to mention all the various combinations of routers and types of connection to the internet. (Or whatever combinations of hardware/software/internet-connection/user-experience/network are relevant to the product in question).

Real users, in the real world, are an invaluable resource for *any* software company, *if* the company chooses to listen to what users tell them about how their product functions in actual use.

Those users can supply tens of thousands of data points, for hundreds or thousands of server configurations - most of which the software developers and testers wouldn't have an opportunity to test. In some cases, engineers wouldn't even imagine the server set-ups some people use in real-life. They can supply data that would be unobtainable in any other way.

(Believe me, in tech support, I've heard some almost unbelievable stories, especially from folks operating in parts of the world far from California experience. People running on batteries, and barely functional phone lines trying to run servers - their issues are radically different from someone with a box living in a server farm).

Users of a software product are the greatest resource a company has to make their product the best available.

The experience of thousands of users is wasted if the engineers at a company are so arrogant and condescending that they respond to user feedback by dismissing the information provided as somehow invalid or incorrect when it doesn't fit their preconceived beliefs or QA tests.

People know that there will be bugs or unanticipated conflicts in all complex software products. And they usually don't mind all that much when they encounter a problem... as long as their reports are welcomed by the tech support department, and they receive some kind of prompt response - an immediate work-around or suggestion, with later follow-up and a permanent fix. Things that let them know that they have been taken seriously!

On the other hand, when they attempt to get advice on resolving a problem, and are ignored, or are deemed too stupid to use the product or are told they couldn't possibly have had the experience they report ("our software wouldn't behave that way! it must be user error..."), most people do tend to strongly resent that kind of response. If this kind of experience is encountered more than a couple of times, that person is likely to look for a different software solution (unless they are a Windows user and have become inured to bugs, abuse and neglect).

Doing technical support, means having the wonderful and irreproducible opportunity to communicate with many users. A good tech support manager can take the aggregated information, and with skill at observing patterns, can report many possible bugs or conflicts to the engineering/QA department. In essence, this means being able to filter and refine information from many sources, to give to engineering a report that is much more useful than a single user's vague complaint on a mailing list, of some poorly defined difficulty...

I have had the good fortune to have acquired a reputation for accuracy in bug reporting very early in my tenure with 3 of the 4 software publishers who have chosen to employ me. The information I supplied to QA was very much appreciated by the engineering departments of those companies.

A small amount of detective work, based on respect and appreciation of the intelligence and observational skills of end users, can lead to a pragmatic solution for the user in question so that she/he can get on with their primary business (which is seldom full-time server management for folks running servers on the Macintosh platform), and ultimately to improvements in the software.

I *know* that if the three people who responded to this thread [about Squid causing trouble] on the WebTen mailing list were directly asked, they will all tell you that they use 3rd party plug-ins with WebTen (in fact, I seem to recall that they all use Lasso). And that is most likely why turning off Squid helped all three of them (and many other people as well). And no, I don't think it is Lasso's fault in particular, since other plug-in combos with Squid are also problematic, but Lasso is probably the #1 plug-in used with WebTen these days, so it will be the one you hear about most often.

In the question of Squid vs. 3rd party-plug-ins, I am not assigning blame - I do not know or care whether or not it is Squid's fault, or the fault of the third party plug-in, or the fault of the way the WebStar API for plug-ins was implemented in WebTen...

None of that matters!!

The only thing that matters is that users are listened to, and a pragmatic solution is found (until the underlying issue is found and resolved or is deemed to be trivial). People need tech support to assist in making their server(s) work for them in accordance with their requirements.

The very customers who buy software products are providing the money to make it possible to create the next release... the very thing that makes it possible for any commercial software product to move forward. A company can try to move forward based on what engineers think would be cool (which may have nothing to do with real-world user's needs), or they can improve from user's feedback...

I believe that user's feedback, truly taken to heart, combined with superior engineering, testing and technical support are the things that can make a product outstanding in it's category.

The user feedback component is an absolutely essential component and cannot be ignored by any software company that hopes to be still be around 5 years from now.

Although 3rd party plug-ins are not relevant to iTools users, the basic concern of how various components might interact with other is a real concern. A test with a few web pages in a QA lab, is not the same as a real server with 250+ virtual hosts, some with IP multi-homing, along with thousands of pages using PHP, iASP and a handful of other technologies... things might all work great or they might not always.

- stephanie
--
+++++++++++++++++++++++++++++++++
Stephanie Wright
lioness@xxxxxxxxxxxxxx
+++++++++++++++++++++++++++++++++