NOTE: This is a VERY long message, as it attempts to describe as many
aspects of the bake-off as possible (including both successes
and failures), and suggests what the PWG should do about it.
What little that has been posted to the list (thus far) regarding the
results of the Bake-Off have been pretty "glowing" with respect to the
overall success of the event.
I was hoping someone else--particularly a participating printer
manufacturer--would be the first to submit comments pointing out the
problems encountered during testing. However, since no such comments
have yet been posted, I'd like to take a shot and "call a spade a
spade" with regard to the real results...as seen from the perspective
of an independent software developer who has developed printer status
monitoring software using the Printer MIB.
Before continuing, I'd like to point out that some folks in the PWG
might feel this message is a bit of a "spoiler" in that it appears to
take some (or all) of the shine off of the bake-off event. Apologies
are extended in advance to those who might perceive this message in
that vein.
This message is meant to inform the general public of the Printer MIB
implementation problems found during the bake-off. Included are some
comments regarding the impact of those implementation problems for
software developers trying to use those implementations with software
destined for the general marketplace.
A total of six printer manufacturers were present, each with a single
printer containing an implementation of the Printer MIB:
Hewlett-Packard LaserJet 5Si
IBM Network Printer 17
Kyocera XS3600+
Lexmark Optra R
Tektronix Phaser Print 550
Xerox Xprint 4920
A total of three software products were used to communicate with these
printers for testing and demonstration purposes:
Genoa General MIB testing facilities
Kyocera Printer Console (Windows)
Underscore SENSE Printer Status Monitor (SunOS Motif)
Genoa brought their software used for general MIB compliance testing.
The results of their work are described in detail later in this message.
Kyocera's software was not actually "demonstrated" to the group as a
whole; instead, the software was demonstrated to individual persons as
interest was shown. This situation was unfortunate, as Kyocera's
software appeared to be quite capable as a general mechanism for
viewing MIB variables and resolving the operating condition of the
printer. I believe this software is only available for the Windows
environment (3.1 and '95?); perhaps Kyocera can comment on the
availability for other platforms.
As a software developer, I found the Kyocera software to be both
attractive (in a practical and usable sense) and functional, although
I did not perform comprehensive apples-and-apples testing on multiple
printers to determine the precise level compatilibity and consistency.
It does appear to be a *very* nice tool, though. Congratulations to
Kyocera for this development!
Underscore conducted the first public demonstration of the SENSE
reference implementation as developed thus far over the last seven
months. Specifically, a GUI status monitor application was shown in
which the status of all available printers was shown in iconic forms
within a "desktop metaphor"; the background color of each icon
reflected the current operating condition of the associated printer,
using the typical "traffic light" metaphor of red, yellow and green
background colors.
Double-clicking an icon would present a "console" in which three
aspects of the printer could be viewed through selection of one of
the available tab cards:
Status Summary General identification and status information
Properties List of sorted MIB variable names and values
Events Chronological list of received printer events
The results of the Underscore SENSE demonstration were a "qualified"
success, in that all activities didn't go quite as expected. In the
"success" column:
- Alerts received from a printer (as forwarded by the SENSE server)
were properly displayed as pop-up informational dialog boxes, with
all known event data displayed in a scrolling list, and the
overall condition highlighted by a reproduction of the printer
icon using the "traffic light" metaphor (ie, red, yellow, green).
- Even if the printer was not in communication with the SENSE server
(via the associated SENSE Publisher process), the user could view
the contents of the Printer MIB as last presented by the SENSE
Publisher. (This is a feature you don't usually get with a
standard SNMP network management station system.)
In the "failure" column:
- Due to a problem in the CMU SNMP code (used as the basis for SNMP
support in the SENSE Publisher), full support for the IBM Network
Printer 17 was not possible due to strange timeout problems when
requesting MIB variables. (This turned out to be a silly little
problem easily fixed in the SENSE software using the CMU code.)
- Not enough tests were performed during the demonstration that
would allow the audience to observe true interoperability.
The burden of the last failure must be shouldered by Underscore, as we
should have been much better prepared to walk through a concise set of
tests that could be applied evenly across all participating printers.
(Underscore is currently in the process of developing such a
comprehensive test suite; hopefully another bake-off in the near
future will provide us the opportunity to better demonstrate the SENSE
software.)
Due to time constraints, Genoa was only able to test and record the
values of the three "magic decoder ring" values in the Host Resources
MIB that, when taken together, resolve the "overall current condition"
of the printer.
Genoa performed a series of tests in which a target printer was placed
into each of the following states and the values of the three HR MIB
variables recorded:
Off Line
Running
Door Open
Paper Tray
Printing
While these particular printer states are indeed important, we should
have also included (IMHO) the following states:
Power Saver Mode (ie, "Energy Star" state)
Paper Jam
Paper Out
Output Bin Full
Toner Low
Toner Out
Granted, testing these last two states (Toner Low/Out) would require
each vendor to bring along toner cartridges that would trigger those
states, but this is not really difficult to do. (We do that all the
time at Underscore; we retain low and empty toner cartridges precisely
for this purpose, and hope vendors do the same during their internal
testing.)
Of the six printers present at the event, only four of them were
actually tested by the Genoa folks. The Hewlett-Packard LaserJet 5Si
was packed up for shipment back to Boise before the Genoa tests got
underway; the Kyocera XS3600+ was not tested for some reason, even
though it was up and running during the testing period (not enough
time, perhaps?).
Of the four printers tested, only TWO of them had the complete set of
tests applied, namely, the IBM Network Printer 17 and the Xerox Xprint
4920. For whatever reason, the Lexmark Optra R and Tektronix
PhaserPrint 550 did not have all the tests applied, including some
rather critical tests. (Why was this???)
As a result, I must honestly submit that this situation--in which only
two of the six available printers were fully tested against a rather
minimal set of state tests--represents a major failure of the
bake-off. As a cooperating group of printer industry players, we
should have knuckled down and worked together as a group to ensure ALL
of the printers were tested in a consistent fashion.
The Genoa folks handed out a sheet summarizing the results of the
tests at the end of the day on Wednesday, July 24. This summary sheet
described the results in a "per-printer" basis, in which sections on
the sheet showed the results of the tests for each printer.
(Hopefully the Genoa folks will follow up on their commitment to
publish those results to the PWG in the near future.)
While the Genoa summary sheet was very insightful regarding the final
results of the testing, a slightly different presentation of the
results are shown below that should help the reader better compare the
actual results between the participating printers.
A separate table is presented for each of the particular printer state
tests, in which the results for all participating printers are shown
in a manner that facilitates rapid comparison of the results between
vendors.
To make the following tables a bit easier to read, the standard HR MIB
object names have been abbreviated as follows:
DevStat == hrDeviceStatus
PrtStat == hrPrinterStatus
PrtError == hrPrinterDetectedErrorState
Note that a value of "n/a" indicates results of the test were not
available for the associated printer. (Again, why did some of the
printers not have results for some of these tests?)
The last column of each table ("PrinterStatus") shows the textual name
of the overall state of the printer, as documented in the "Overall
Printer Status" section (2.2.12.2) of RFC 1759. It is this value that
ultimately states whether the printer is accurately portraying the
current operational condition such that an external management
application is able to convey the proper, consistent state to the
user.
Note that a value of "--Not Defined--" in the "PrinterStatus" column
indicates that the combinations of the three HR MIB variable values DO
NOT MAP to one of the textual states as described in RFC 1759. It
should be generally understood that anytime this value appears in any
of the tables, then EITHER the corresponding vendor has a PROBLEM that
needs to be resolved in that printer product, OR the PWG must work to
refine the definitions of those combinatorial values for the sake of
consistency and clarity.
To assist the reader in digesting this information and drawing
whatever conclusions are appropriate, I have attached an extract of
RFC 1759 to this message that describes the HR MIB "magic decoder
ring" table of combinatorial values, and the definitions of those MIB
objects (with more detailed value definitions).
Following are the tabular results of the tests; after each table I
have included a summary of the results (from Underscore's point of
view). Note that all numeric values in these tables are in decimal
(whereas Genoa's summary sheet shows these values in hex):
Test: Off Line
DevStat PrtStat PrtError PrinterStatus
IBM 5 1 64 --Not Defined--
Lexmark 5 1 2 Off-line
Tektronix n/a n/a n/a
Xerox 2 1 2 --Not Defined--
Summary:
I would suggest that this test was an unqualified FAILURE:
- Only Lexmark got this one right.
- IBM has the bit order for hrPrinterDetectedErrorState in
reverse order; they show values in which bit #0 is the LEAST
significant bit, when RFC 1759 clearly states that bit #0 is
the MOST significant bit. This problem ultimately hurts their
results for most other tests.
- Xerox has more of a problem on its hands, as the set of HR MIB
values does not properly map to a standard "Printer Status" value.
Test: Running
DevStat PrtStat PrtError PrinterStatus
IBM 2 3 0 Normal
Lexmark 2 3 0 Normal
Tektronix n/a n/a n/a
Xerox 2 3 0 Normal
Summary:
I would suggest that this test was an unqualified SUCCESS.
Despite the missing values for Tektronix, this test showed
uniform, accurate results from all other players. (I'll bet that
Tektronix got this one right... ;-)
Test: Door Open
DevStat PrtStat PrtError PrinterStatus
IBM 5 1 16 Critical Alert Active
Lexmark 5 1 10 Critical Alert Active
Tektronix 5 1 1 Critical Alert Active
Xerox 3 3 1 Non Critical Alert Active
Summary:
I would suggest that this test was a qualified FAILURE. Most
players got most of it right in that the hrDeviceStatus and
hrPrinterStatus variables should have the values 5 and 1,
respectively. However, too many other problems are present that
keep this test from being considered a success:
- Lexmark came closest to portraying the proper results; their
value for the hrPrinterDetectedErrorState clearly indicates
"doorOpen", but for some reason the printer also declared
itself as being "offline" at the same time (was this a fluke,
or does the printer always go "offline" in this scenario?).
- IBM would have been dead-on with their values, but the
unfortunate reversal of bit order in hrPrinterDetectedErrorState
results in an incorrect resolution of the actual problem; once
the bit order problem is resolved, they should be in compliance.
- Tektronix *almost* got this one right, but instead of showing
"doorOpen", they indicate "serviceRequested" for the
hrPrinterDetectedErrorState variable. This anomoly could
cause heartburn in some management applications (such as those
from Underscore), as "serviceRequested" would usually map to a
situation where a "technician" would be required to resolve
the problem (and not a simple "user").
- Xerox presents an interesting problem with its results. The
values were *almost* accurate, except that the net results
indicate a NON critical alert. (Is this accurate? If so,
then printing would have continued even if the door was open;
however, this is not typical for printers of this class within
the industry! Comments from Xerox?)
Test: Paper Tray
DevStat PrtStat PrtError PrinterStatus
IBM 5 1 1 Critical Alert Active
Lexmark n/a n/a n/a
Tektronix 5 1 1 Critical Alert Active
Xerox 3 4 1 d
Summary:
I would suggest that this test was a qualified FAILURE, but it's
really hard to tell since I'm not sure whether Genoa's test
scenario involved pulling the input while a job was printing (and
the job required that tray), or whether the printer was idle and
the tray was simply pulled out. (Can someone comment on this
situation?)
NOTE: This particular test needs to be seriously discussed by the
PWG to further clarify the intended correct results. This
situation is one of many in which the set of defined values
for hrPrinterDetectedErrorState is seriously LACKING with
respect to the common problem states seen in printers of
this class (ie, network laser printers). In particular,
there are no values for "input tray missing" or for
"output bin full" (my biggest complaint!).
Despite the ambiguity of the test scenario, too many other
problems are present that keep this test from being considered a
success:
- Tektronix was the only player to get this one right, but this
may be hard to declare, since we don't know if the paper tray
was pulled while a job was present and needed the tray
(CRITICAL alert), or while the printer was idle (NON critical).
- Xerox again indicates a NON critical alert for this state, and
it's hard to tell whether they are right or wrong, based on
the same statements made for the Tektronix results.
- IBM appears to have gotten this right, but since they have
the bit order wrong for hrPrinterDetectedErrorState, the
results are not good, since the proper bit order would have
them declaring "lowPaper" instead of "serviceRequested".
- What happened to the Lexmark test values?? (Very strange.)
Test: Printing
DevStat PrtStat PrtError PrinterStatus
IBM 2 4 0 Busy
Lexmark n/a n/a n/a
Tektronix 2 4 0 Busy
Xerox 2 4 0 Busy
Summary:
I would suggest that this test was an unqualified SUCCESS.
Despite the missing values for Lexmark, this test showed
uniform, accurate results from all other players. (I'll bet that
Lexmark got this one right... ;-)
Now let's take a look at the summaries of the tests:
Test Name Results
--------- -------
Off Line Failure
Running Success
Door Open Failure
Paper Tray Failure
Printing Success
So what can we conclude from these results? Can the bake-off be
considered a success based on these limited "formal" tests?
With all due respect to my many esteemed colleagues, I must say NO,
the first PWG bake-off can NOT be considered a success. I carefully
make this statement based on these observations:
- Of the six printers brought to the event, only four were actually
tested in a relatively consistent environment.
- Of the four printers tested, only two printers had results for all
defined tests; hence, when the dust settles, only 33% of the
printer participants were actually involved in a useful bake-off
scenario.
- The only "successful" tests were those in which the printers were
NOT in a problem state (ie, "Running" or "Printing"). I would bet
that our customers would not find this very amusing, as printer
problem identification and notification is one of the two core
reasons why our customers demand SNMP-like capabilities in
printers. (Acquiring printer capabilities and related
configuration being the other core reason.)
- The "Paper Tray" test *might* have been considered a success, but
even then only ONE player got this test right--either Tektronix
OR Xerox, depending on the precise nature of the test scenario.
- In general, the game plan for the bake-off was not well defined in
advance so that more complete and comprehensive testing could be
performed (and I must share in the blame for this failure). In
particular, we needed to have *many* more tests that involve
changes to the Alert Table.
Ira McDonald (Xerox consultant and personal "pen pal" of mine) wrote
in a recent message:
> I'm fascinated to hear what you thought of the 'bake-off' event. I
> suspect that the apparent variability of implementation choices for the
> 'hrDeviceStatus', 'hrPrinterStatus', and 'hrPrinterDetectedErrorState'
> objects has you concerned.
You BET I'm concerned about this problem. As an independent software
vendor attempting to produce and market printer management products
based on the Printer MIB, Underscore is extremely concerned about this
situation. After all, if the HR MIB magic decoder ring values aren't
right, then what's the point of having them in the first place?
We need to know whether or not the HR MIB values are indeed going to
represent the overall condition of the target printer (with respect to
the "traffic light" metaphor that customers have come to expect). If
vendors are going to be "all over the place" on these values, then
Underscore will have to completely ignore those values, instead
relying on the cummulative state of the Alert Table. (ugh!)
I realize that many folks would like to paint a rosy picture of the
PWG bake-off event with statements such as:
> But in another sense, the 'bake-off' has to
> be considered a 'success', because a number of vendors brought live stuff
> together into one room and made a real effort to understand each other's
> implementation choices.
With all due respect to the PWG at large, let's not find ourselves
patting each other on the back just because we all got together in a
room for a couple of days with our respective products. Paraphrasing
a statement made at the end of the bake-off (regarding the viability
of having an official PWG bake-off at Networld-Interop next month):
"This just isn't 'news' to the outside world."
I don't know about you, but the credibility of the PWG as a recognized
industry trade organization is of paramount concern for both myself
and my company, particularly considering the exhorbitant costs for
maintaining participation at the many far-flung meetings held over the
last three years (September 1993 being the first real PWG meeting).
One disappointing aspect was that we DIDN'T do what was stated at the
end of that last quote above, namely:
"...a number of vendors...made a real effort to understand each
other's implementation choices."
Unless I was out of the room at the time, I don't recall us sitting
down and going through the Genoa results and asking each other why the
results were so inconsistent. (Was this done? Was I out of the room
at the time? If so, then someone please post the results of that
discussion as soon as possible.) If such a candid and honest
post-test examination was indeed conducted, then the results of that
discussion should be publicly posted for all to see, IMHO.
In closing, I'd like to suggest that more interoperability testing
must be performed in the very near future. If the PWG is unable to
conduct another bake-off, then Underscore would be more than willing
to arrange for more comprehensive testing of printer products in its
offices over the course of the next few months.
Please make no mistake: consistent AND accurate SNMP Printer MIB
implementations are at the heart of Underscore's (and other software
vendors') printer management application products. If we can't get
this right, then we essentially have no product to sell. With that
said, Underscore STRONGLY ENCOURAGES all printer vendors to contact us
to arrange for compatibility testing of their products as soon as
possible...even if another PWG Bake-Off is ultimately arranged in the
future.
Underscore needs to ship product as soon as humanly possible, but the
current state of Printer MIB conformance and compatibility represents
a critical roadblock to that goal. For a small company like ours, a
failure in the marketplace for an initial product launch could be
incredibly disastrous...and that's just what we're facing given the
current state of compatibility for vendors incorporating the Printer
MIB in their printers.
As always, comments from other folks are encouraged, particularly
those with differing views on whether the first PWG bake-off event was
a success or not.
...jay
----------------------------------------------------------------------
-- JK Martin | Internet: jkm@underscore.com --
-- Underscore, Inc. | Usenet: ..!uunet!mv.com!uscore!jkm --
-- 41C Sagamore Park Road | Voice: (603) 889-7000 --
-- Hudson, NH 03051-4915 | Fax: (603) 889-2699 --
----------------------------------------------------------------------
RFC 1759 Extracts
2.2.13.2. Overall Printer Status
Of the many states a printer can be in, certain states are more
"interesting" because of the distinct actions they are likely to
provoke in the administrator. These states may be applied to the
printer as a whole, or to a particular sub-unit of the printer.
These named states are:
Non Critical Alert Active - For the printer this means that one or
more sub-units have a non-critical alert active. For a sub-unit,
this means that the sub-unit has a non-critical alert active.
Critical Alert Active - For the printer this means that one or more
sub-units have a critical alert active. For a sub-unit, this means
that the sub-unit has a critical alert active.
Unavailable - The printer or sub-unit is unavailable for use (this is
the same as "broken" or "down" in other terminologies). A trained
service person is typically necessary to make it available.
Busy / Temporarily Unavailable - The printer or sub-unit is
operational but currently occupied with a request for activity. The
sub-unit will become available without the need of human interaction.
Moving on-line or off-line - The printer is either off-line, in the
process of moving off-line or in the process of moving back on-line;
for example on high end printers reloading paper involves a
transition to off-line to open the paper bin, it is then filled and,
finally, there is a transition back to on-line as the paper bin is
repositioned for printing.
Standby - The printer or sub-unit is unavailable for use because it
is partially powered down and may need some period of time to become
fully operational again. A unit in Standby state shall respond to
network management requests.
The Host MIB provides three status objects that can be used to
describe the status of a printer: (1) hrDeviceStatus in the entry in
the Host MIB hrDeviceTable; (2) hrPrinterStatus in the
hrPrinterTable; and (3) hrPrinterDetectedErrorState in the
hrPrinterTable. These objects describe many of the states that a
printer can be in. The following table shows how the "interesting"
states named above can be recognized by inspecting the values of the
three printer-related objects in the Host MIB:
Printer hrDeviceStatus hrPrinterStatus hrPrinterDetectedErrorState
Status
Normal running(2) idle(3) none set
Busy/ running(2) printing(4)
Temporarily
Unavailable
Non Critical warning(3) idle(3) or could be: lowPaper,
Alert Active printing(4) lowToner, or
serviceRequested
Critical down(5) other(1) could be: jammed,
Alert Active noPaper, noToner,
coverOpen, or
serviceRequested
Unavailable down(5) other(1)
Moving off- warning(3) idle(3) or offline
line printing(4)
Off-line down(5) other(1) offline
Moving down(5) warmup(5)
on-line
Standby running(2) other(1)
These named states are only a subset of the possible states - they
are not an exhaustive list of the possible states. Nevertheless,
several things should be noted. When using these states, it is not
possible to detect when both critical and non-critical alerts are
pending - if both are pending, the Critical Alert Active state will
prevail. In addition, a printer in the Standby state will be
represented in the Host MIB with a device status of running(2) and a
printer status of other(1), a set of states that don't uniquely
distinguish this important printer state.
Although the above mapping is workable, it would be improved with a
few additions to hrDeviceStatus and hrPrinterStatus in the Host
Resources MIB. In particular, it would be appropriate to add a
"standby" enumeration to hrDeviceStatus. Similarly, it would be
useful to add the following states to hrPrinterStatus: "offline" to
indicate that reason for the printer being down (instead of having to
use "other") which allows both "warning" and "offline" to indicate
going offline and "down" and "offline" to indicate offline and
"notApplicable" to cover cases, such as "standby", where the device
state completely describes the state of the device.
Detailed status per sub-unit is reported in the sub-unit status
fields.
2.2.13.2.1. Host MIB Printer Status
For completeness, the definitions of the Printer Status objects of
the Host MIB are given below:
hrDeviceStatus OBJECT-TYPE
SYNTAX INTEGER {
unknown(1),
running(2),
warning(3),
testing(4),
down(5)
}
ACCESS read-only
STATUS mandatory
DESCRIPTION
"The current operational state of the device
described by this row of the table. A value
unknown(1) indicates that the current state of the
device is unknown. running(2) indicates that the
device is up and running and that no unusual error
conditions are known. The warning(3) state
indicates that agent has been informed of an
unusual error condition by the operational software
(e.g., a disk device driver) but that the device is
still 'operational'. An example would be high
number of soft errors on a disk. A value of
testing(4), indicates that the device is not
available for use because it is in the testing
state. The state of down(5) is used only when the
agent has been informed that the device is not
available for any use."
::= { hrDeviceEntry 5 }
hrPrinterStatus OBJECT-TYPE
SYNTAX INTEGER {
other(1),
unknown(2),
idle(3),
printing(4),
warmup(5)
}
ACCESS read-only
STATUS mandatory
DESCRIPTION
"The current status of this printer device. When
in the idle(1), printing(2), or warmup(3) state,
the corresponding hrDeviceStatus should be
running(2) or warning(3). When in the unknown
state, the corresponding hrDeviceStatus should be
unknown(1)."
::= { hrPrinterEntry 1 }
hrPrinterDetectedErrorState OBJECT-TYPE
SYNTAX OCTET STRING
ACCESS read-only
STATUS mandatory
DESCRIPTION
"This object represents any error conditions
detected by the printer. The error conditions are
encoded as bits in an octet string, with the
following definitions:
Condition Bit # hrDeviceStatus
lowPaper 0 warning(3)
noPaper 1 down(5)
lowToner 2 warning(3)
noToner 3 down(5)
doorOpen 4 down(5)
jammed 5 down(5)
offline 6 down(5)
serviceRequested 7 warning(3)
If multiple conditions are currently detected and
the hrDeviceStatus would not otherwise be
unknown(1) or testing(4), the hrDeviceStatus shall
correspond to the worst state of those indicated,
where down(5) is worse than warning(3) which is
worse than running(2).
Bits are numbered starting with the most
significant bit of the first byte being bit 0, the
least significant bit of the first byte being bit
7, the most significant bit of the second byte
being bit 8, and so on. A one bit encodes that
the condition was detected, while a zero bit
encodes that the condition was not detected.
This object is useful for alerting an operator to
specific warning or error conditions that may
occur, especially those requiring human
intervention."
::= { hrPrinterEntry 2 }
# # # # #