[LISPWORKS][Common Lisp HyperSpec (TM)] [Previous][Up][Next]


Issue DOTIMES-IGNORE Writeup

Issue:        DOTIMES-IGNORE

Forum: Cleanup

References: DOLIST (p126), DOTIMES (p126-128), IGNORE (p160),

LOOP (X3J13/89-004), DO-SYMBOLS (p187), DO-ALL-SYMBOLS (p188),

DO-EXTERNAL-SYMBOLS (p187), DO (pp122-126), DO* (pp122-126)

Category: CLARIFICATION

Edit history: 20-Jul-90, Version 1 by Pitman

06-Mar-91, Version 2 by Pitman

15-Mar-91, Version 3 by Pitman

Status: For Internal Discussion

Problem Description:

The IGNORE declaration is described in terms of whether a variable is

`used'. But what constitutes a use? Several iteration primitives are

described as macros, and the nature of the expansion is not well

enough specified for the answer to this question to be easily

determined.

Consider:

(DOTIMES (A 3) (PRINT 'FOO))

Is A used? Would it be appropriate to write:

(DOTIMES (A 3) (DECLARE (IGNORE A)) (PRINT 'FOO))

In some implementations, this would seem to be desirable, while in

other implementations it would seem undesirable.

In some implementations, such as an older Genera release, when the

compiler sees an IGNORE declaration, it `commits' the fact that the

variable will not be used and sometimes cannot later back out and

produce a successful compilation when this assumption is later realized

to be wrong in the process of code-walking. Because such a program

is technically in error, this is a conforming situation. But it

means that the choice of whether to use IGNORE or not has potential

semantic impact.

This issue started out as being just about DOTIMES (hence the name)

but has been generalized to include all CL iteration forms.

Terminology:

For the purposes of this issue, the following terminology applies:

An "explicit use of a variable in a form" is a use that would be

apparent in the normal semantics of the form. That is, if all

subforms of that form were macroexpanded, but the form itself were

not. [This isn't how an implementation has to detect an explicit

form; it's just a definition of what explicit use means. The intent

is that an explicit use be one which does not expose any undocumented

details of the form's macro expansion (if the form is in fact a macro).]

An "iteration form" means a DOLIST, DOTIMES, LOOP, DO-SYMBOLS,

DO-ALL-SYMBOLS, DO-EXTERNAL-SYMBOLS, DO, or DO* form.

An "iteration variable" is a variable which is "explicitly used"

as a variable to be bound by by an "iteration form".

Voting Instructions:

Proposals which have a "+" in their name are compatible with any

other proposals, and may be voted in addition to the other proposals.

Proposal (DOTIMES-IGNORE:NEVER):

Clarify that iteration variables are, by definition, always `used'.

That is, clarify that using

(DECLARE (IGNORE ...))

for such an iteration variable has undefined consequences.

Rationale:

This is analagous to the situation in DEFMETHOD where a specialized

required argument is considered a use of the argument.

Example:

;; The following would be correct and should not be warned about.

(DOTIMES (I 5)

(PRINT 'FOO))

;; The following would not be correct and might be warned about.

(DOTIMES (I 5)

(DECLARE (IGNORE I))

(PRINT 'FOO))

Proposal (DOTIMES-IGNORE:UNLESS-EXPLICIT):

Clarify that unless an variable in an iteration form is explicitly

used in the form which binds it, it might be considered "unused" by

the implementation, and that it is always permissible to use

(DECLARE (IGNORE ...))

for such a unused variable.

Rationale:

This doesn't force users to think about the macro expansion and tends

to treat iteration variables more like variables bound by LET, LET*,

and LAMBDA.

Example:

;; The following would be correct and should not be warned about.

(DOTIMES (I 5)

(DECLARE (IGNORE I))

(PRINT 'FOO))

;; The following would not be correct and might be warned about.

(DOTIMES (I 5)

(PRINT 'FOO))

Proposal (DOTIMES-IGNORE:INVISIBLE-REFERENCES):

Introduce a macro

WITH-INVISIBLE-REFERENCES ({vars}*) {forms}* [Macro]

Within the context of this macro, any uses of the <vars> are not

regarded as `uses' of the <vars> for the purposes of compiler warnings.

Clarify that macros like DOTIMES which expand into code which do

invisible computations on user-supplied variables must do the equivalent

of wrapping WITH-INVISIBLE-REFERNCES around those invisible uses, so that

those uses will not be counted as uses.

Rationale:

This makes the mechanism for invisible uses available to users so that they

can write forms like DOTIMES which manipulate user-supplied variables

and yet warn about lack of explicit use of those variables.

Example:

;; The following would be correct and should not be warned about.

(DOTIMES (I 5)

(DECLARE (IGNORE I))

(PRINT 'FOO))

;; The following would not be correct and might be warned about.

(DOTIMES (I 5)

(PRINT 'FOO))

;; The following would be correct. User might be warned that X

;; is not used. (This is a pathological case which wouldn't be a

;; normal use. For a `real' use see the expansion of DOTIMES in

;; the Symbolics Genera current practice.)

(LET ((X 3))

(WITH-INVISIBLE-REFERENCES (X) (INCF X) (PRINT X))

0)

4

=> 0

Proposal (DOTIMES-IGNORE:+IGNORE-FUNCTION):

Introduce a function

IGNORE &REST IGNORE [Function]

Accepts any number of arguments and returns NIL.

Function calls to IGNORE can effectively be optimized away by a compiler,

except where side-effects might occur. That is, (IGNORE X #'Y (Z)) is

equivalent to (PROGN X #'Y (Z) NIL), but is more perspicuous about the

user's intent to identify ignored items.

Rationale:

This leaves the original problem in place but provides a compromise

that people can learn to use. It also has the side benefit of providing

an obvious syntax for `ignorable' variables introduced by some macros.

It also provides an obvious syntax for ignoring FLET'd items.

Example:

;; The following might or might not be correct and might be warned about.

(DOTIMES (I 5)

(DECLARE (IGNORE I))

(PRINT 'FOO))

(DOTIMES (I 5)

(PRINT 'FOO))

;; The following would not be warned about.

(DOTIMES (I 5)

(IGNORE I)

(PRINT 'FOO))

Proposal (DOTIMES-IGNORE:+STYLE-WARNING):

Clarify that any `unused variable' warning must be a STYLE-WARNING, and

may not affect program semantics.

Rationale:

This is similar to STATUS-QUO, but at least insures that the issue has

no semantic impact on code.

Example:

;; The following might or might not be correct and might be warned about

;; but could be suppressed in a context where style warnings were

;; suppressable. In either case, correct code would be generated.

(DOTIMES (I 5)

(DECLARE (IGNORE I))

(PRINT 'FOO))

(DOTIMES (I 5)

(PRINT 'FOO))

Proposal (DOTIMES-IGNORE:+NEW-DECLARATION):

Introduce a new declaration, IGNORABLE, similar to IGNORE, except that

it means that the variable might or might not be used, but that in

neither case should a warning be issued.

Rationale:

A lot of people have asked for this. e.g., it would be useful for

places where variables pop out of nowhere due to the presence of some

macro. e.g., in New Flavors (think of it as a user program from the

point of view of this proposal) the variable SELF magically appears in a

DEFMETHOD. The definition of FLAVOR:DEFMETHOD might, therefore, want

to declare SELF to be IGNORABLE since not all Flavors' DEFMETHOD forms

actually use this.

Example:

Given...

(DEFMACRO WITH-FOO (X &BODY STUFF)

`(LET ((FOO (FROB ,X)))

(DECLARE (IGNORABLE FOO))

,@STUFF))

Neither of the following two uses would be warned about:

(WITH-FOO (F) (G))

or (WITH-FOO (F) (H FOO))

Proposal (DOTIMES-IGNORE:+FUNCTION-DECLARATIONS):

Permit the IGNORE (and IGNORABLE, if it passes--see proposal

+NEW-DECLARATION) declarations to contain references to #'name

in order to refer to a function name instead of a variable name.

Rationale:

This would be useful, for example, in the expansion of DEFMETHOD

in order to declare that CALL-NEXT-METHOD and NEXT-METHOD-P might

or might not be used, but that no warning should be produced in

either case. Some user programs presumably have similar needs.

Example:

(DEFMACRO DEFMETHOD ...

`(... (FLET ((CALL-NEXT-METHOD ...)

(NEXT-METHOD-P ...))

(DECLARE (IGNORABLE #'CALL-NEXT-METHOD

#'NEXT-METHOD-P))

...)))

Proposal (DOTIMES-IGNORE:STATUS-QUO):

Accept the fact that the use of (DECLARE (IGNORE ...)) for an iteration

variable will always cause a warning in some implementations and that

the failure to use it will always cause a warning in some other

implementations. The net effect of this will be that no code which uses

any iteration primitive, DO, LOOP, or otherwise without using all iteration

variables explicitly will be free from gratuitous whining of at least

some compilers. Since the presence of

(DECLARE (IGNORE ...))

is sometimes fatal to some compilers, people will learn to live without it,

or will introduce creative other ways to `use' the variable to muffle the

warnings, sometimes at a performance penalty.

Rationale:

Implements the status quo.

Example:

;; The following might or might not be correct and might be warned about.

(DOTIMES (I 5)

(DECLARE (IGNORE I))

(PRINT 'FOO))

(DOTIMES (I 5)

(PRINT 'FOO))

Test Case:

(DEFUN COMPILER-WARNINGS-P (BODIES &OPTIONAL (FLAG NIL))

(MAPCAR #'(LAMBDA (BODY)

(FLET ((TRY (BODY)

(LET ((WARNING

(WITH-OUTPUT-TO-STRING (*ERROR-OUTPUT*)

(COMPILE NIL `(LAMBDA ,@BODY)))))

(IF FLAG WARNING (> (LENGTH WARNING) 0)))))

(LIST (TRY BODY)

(TRY (LIST (CAR BODY)

(REMOVE-IF

#'(LAMBDA (X)

(AND (NOT (ATOM X))

(EQ (CAR X) 'DECLARE)))

(CADR BODY)))))))

BODIES))

(DEFUN TEST-CASE ()

(COMPILER-WARNINGS-P

'(((X) (DOTIMES (A X) (DECLARE (IGNORE A)) (PRINT 'FOO)))

((X) (DOLIST (A X) (DECLARE (IGNORE A)) (PRINT 'FOO)))

((X) (DO-SYMBOLS (S X) (DECLARE (IGNORE S)) (PRINT 'FOO)))

((X) (DO-EXTERNAL-SYMBOLS (S X) (DECLARE (IGNORE S)) (PRINT 'FOO)))

(() (DO-ALL-SYMBOLS (S) (DECLARE (IGNORE S)) (PRINT 'FOO)))

((X) (LOOP FOR I IN X DO (DECLARE (IGNORE I)) (PRINT 'FOO)))

((X) (LOOP FOR L ON X DO (DECLARE (IGNORE L)) (PRINT 'FOO)))

((X) (DO ((L X X)) (NIL) (DECLARE (IGNORE L)) (PRINT 'FOO)))

((X) (DO* ((L X X)) (NIL) (DECLARE (IGNORE L)) (PRINT 'FOO))))))

Current Practice:

Symbolics Genera implements INVISIBLE-REFERENCES (except that the

macro is called COMPILER:INVISIBLE-REFERENCES) and by implication

it implements UNLESS-EXPLICIT and STATUS-QUO. It also implements

+IGNORE-FUNCTION, and probably implements +STYLE-WARNING.

Test case results...

Symbolics Genera 8.0.1: ((NIL T) (T T ) (T NIL) (T NIL) (T NIL)

(T T) (T T ) (T T ) (T T ))

Symbolics Genera 8.1: ((NIL T) (NIL T) (T NIL) (T NIL) (T NIL)

(T T) (T T) (T T ) (T T ))

Symbolics Cloe: ((NIL T) (T T ) (NIL NIL) (T NIL) (T T )

(T T) (T NIL) (T T ) (T T ))

In Symbolics Genera, the macroexpansion of (DOTIMES (A 3) (PRINT 'FOO)) is:

(PROG ((A 0))

#:G1689 (PRINT 'FOO)

(COMPILER:INVISIBLE-REFERENCES (A)

(IF (PROGN (SETQ A (1+ A))

(< A 3))

(PROGN (GO #:G1689))))

(RETURN NIL))

Cost to Implementors:

NEVER: Relatively small. It's straightforward for macros to add a few

spurious additional references in order to assure that no unused warnings

occur.

UNLESS-EXPLICIT: Medium. This is similar to INVISIBLE-REFERENCES, but

it allows for alternate implementations with equivalent effect because

it doesn't publish the interface.

+IGNORE-FUNCTION: Relatively small.

+NEW-DECLARATION: None to medium. This involves some compiler work,

the amount of which may vary, for implementations that do enough

bookkeeping for it to matter. Implementations which don't ever warn

can just ignore this, of course.

+FUNCTION-DECLARATIONS: Probably relatively small.

INVISIBLE-REFERENCES: None to Medium. This probably involves some

compiler hacking for implementations who care to do this kind of

warning. Strictly, an implementation could simply never warn, and then

this would be free, but some implementations have already established

a precedent of warning, so their customers would probably insist that

they did the work to make warnings continue to work in places where

they were called for.

STATUS-QUO: None.

Cost to Users:

NEVER: Relatively small.

UNLESS-EXPLICIT: Relatively small.

+IGNORE-FUNCTION: Relatively small.

+NEW-DECLARATION: None.

+FUNCTION-DECLARATIONS: None.

INVISIBLE-REFERENCES: Relatively small.

STATUS-QUO: Potentially large nuisance value and medium amount of work

for some users.

When porting to a new platform, you get a lot of warnings. Sometimes

the warnings are useful, sometimes they are not. The worst part is

if they are not useful and you just want to ignore them but you can't

because there are other warnings you want to see and you don't have the

ability to turn off just these warnings. So over and over you have to

wade through these warnings just to get to the others.

Cost of Non-Adoption:

See cost of option STATUS-QUO.

Benefits:

Users don't have to guess about important aspects of language semantics.

Aesthetics:

Proposals NEVER, UNLESS-EXPLICIT, +IGNORE-FUNCTION, +NEW-DECLARATION,

+FUNCTION-DECLARATIONS, and INVISIBLE-REFERENCES definitely improve the

aesthetics in the sense that they give the user the power to make the

code mean a particular thing, rather than leaving it to the discretion

of the implementation what the user meant.

Proposal STATUS-QUO is not aesthetic.

Discussion:

Pitman thinks that anything but STATUS-QUO is livable, but has a

preference for either INVISIBLE-REFERENCES or NEVER. He thinks

any of +IGNORE-FUNCTION, +NEW-DECLARATION, and +FUNCTION-DECLARATIONS

would be nice additions, too.

Dalton is on record as opposing a WITH-INVISIBLE-REFERENCES macro

and an IGNORABLE declaration (see below), but he hasn't seen this

specific proposal (which admittedly isn't likely to change his mind).

Several Symbolics customers have complained about portability

problems due to this lack of specificity.

Gregor Kiczales says: ``in writing PCL, this IGNORE issue was a

major pain in the ass. I never really was able to get it right,

and it ended up showing through to the users.''

JonL White said of mail from Dalton in a much longer discussion:

``he claims that DECLARE-IGNORE is *not* just friendly,

optional advice to the compiler, but assertions about

the program. I may disagree with him, but I can certainly

sympathesize with his viewpoint. This issue in general

-- what *must* declarations do -- is a serious one.''

Margolin said ``We're past the deadline for making significant changes

in the language. We're trying to get the standard edited now. At this

time we shouldn't be making changes unless they fix real language

bugs. I personally don't feel that a style warning is sufficient

justification for a change to the semantics of DOTIMES.'' RWK

responded ``I feel that the style warnings *ARE* an important enough

issue, because these style warnings are by default dictating the

variable semantics, but in a manner inconsistent from implementation

to implementation.''

Some people reviewing version 1 made suggestions which are not

pursued in this version, but which are worthy of note:

- GLS suggested we might need a REPEAT macro, such that

(REPEAT n . body) meant the same as (DOTIMES (I n) . body).

This would paper over the DOTIMES problem, but would still

leave other similar problems unattended to. This angle was

therefore not pursued.

- Barrett and others raised the issue of what counts as a use.

Both reading and writing it, or just reading? Technically,

that's an orthogonal issue, although it has obvious interplay

with anything decided here.

- There was discussion by Dalton, Margolin, and others of changing

the semantics of IGNORE to mean `ignorable'. Some felt this

would be visually confusing. Some felt we should rename IGNORE

to IGNORABLE. Some felt we should introduce IGNORABLE in

addition to IGNORE. [An IGNORABLE declaration was added as an

option in v3 of this proposal. -kmp]

- Jeff Barnett (at Northrop) suggested that we should reconsider

the idea of dignifying a certain variable, IGNORE, as implicitly

ignored and then permit (DOTIMES (IGNORE 5) ...). He observed

that this also works for (MULTIPLE-VALUE-BIND (X IGNORE Z) ...).

This was already brought up as issue IGNORE-VARIABLE (tabled by

Masinter at Mar-89 meeting due to deadline constraints).

Fred White (at BBN) suggested that it should be possible to do

(PROCLAIM '(IGNORE IGNORE)). Pitman pointed out that

(PROCLAIM '(SPECIAL IGNORE)) almost works.

Neil Goldman (at ISI) raised the issue of

(mapcar #'(lambda (V) t) l)

which again calls for the IGNORE variable.

- Eliot@CS.UMASS.EDU pointed out that he just tries to avoid

(DECLARE (IGNORE X)) in favor of just X at toplevel of a body.

Don Cohen (at ISI) disagreed that this was adequate, saying that

he felt that an implementor would be justified in finding such

a `use' to be gratuitous and still warning. (Pitman disagrees

with this disagreement, because sometimes macros expand into

such things and getting a use that is `more significant' might be

a data abstraction violation.) In any case, the use Eliot was

talking about really implements IGNORABLE, not IGNORE.

- There was some exploration into the area of whether DOTIMES should

create a new binding every time. RPG cited that the current

definition of DOTIMES is unintuitive to users of parallel lisps.

Dalton pointed out that if a new binding were made every time, then

the internal variable which DOTIMES stepped could be a different

one, and hence the user's variable would be unused as a matter of

course (i.e., without WITH-INVISIBLE-REFERENCES) since no hidden

operations on it would be needed. Bob Kerns (RWK at ILA) said that

if people were intending to close over an iteration variable, their

intent would still be clearer if they wrote

(DOTIMES (I N) (LET ((I I)) (SPAWN #'(LAMBDA () ...I...))))

rather than just

(DOTIMES (I N) (SPAWN #'(LAMBDA () ...I...)))

- Dalton raised the question of whether all this trouble was really

worth it and whether we shouldn't just specify the expansion of

DOTIMES precisely so users could just know what it would have going

on inside it. RPG said ``Using possible macro expansions to reason

about language design is poor methodology. DOTIMES should be

specified and its implementation choices outlined.'' Barmar

defended (as an example) the Symbolics implementation which

expands differently depending on the situation and worried that

optimizations would be harder if the expansion was dictated. The

idea of showing a space of possible expansions was also discussed

by RWK and others. Barmar said ``By specifying each

operator as independently as possible we have less trouble with

strange interactions.'' Dalton responded ``Except where we don't,

as with DOTIMES and IGNORE. Moreover, it is at least arguable that

when too much is separate the language is more complex and harder

to learn.''

- Neil Goldman cited three examples of problem situations--

- 1) in lambda lists, because a function is used in a role

where it may be passed arguments it does not need.

2) multiple-value-setq, where the values that need to be

consumed are not just an initial sequence of those returned.

3) macros (like DOTIMES), that require the provision of a

variable name that will be lexically bound in the expansion

and permit IGNORE declarations about it.

He suggested that the form of destructuring used by LOOP would

solve these problems. e.g.,

(defun arg1-is-integer (arg1 nil) ...) ;;second arg is not consumed

(defun CAR-is-integer ((car . nil)) ...)

;; first arg must be a CONS. Its CDR is not consumed

(multiple-value-setq ((onea oneb) nil three) ...)

(dotimes (nil n) (push 0 l))

Peter Norvig (at Berkeley) worried about ambiguities in this notation.

(defun optional-CAR-is-integer (&optional (car nil)) ...)

(defmethod CAR-is-integer ((car cons)) ...)

RWK wrote one particular message which Moon identified in mail as the

most important message in the long conversation and which Moon said he

hoped wouldn't get lost. GLS sent mail saying he agreed. The

message was long, but it was a good message, so KMP has picked some choice

paragraphs from it and reproduced them here. These are from the

message of 26 Jul 90 21:26 EDT from RWK@FUJI.ILA.Dialnet.Symbolics.COM:

``Variable semantics is a very fundamental aspect of the language,

and we have failed to specify an important part of it. Traditionally,

we have dismissed "style warnings" as being an "environment" issue.

However, in this case, I do not think it should be so dismissed.

Currently, we have implementations of CL which vocally complain about

opposite usages. Having bogus style warnings go off in this way is

a serious problem for people porting code, because it obscures

legitimate problems. Bogus warnings are a serious waste of time for

porters of code, and they also make venders of portable code look as

if they are careless programmers.

``I strongly feel that a well-written portable Common Lisp program

should compile with no warnings in any well-done Common Lisp

implementation. Otherwise, how is the user of a portable program

supposed to know if the warnings warn of actual problems, or are

just noise.

``I strongly feel that X3J13 should firmly nail down under what

circumstances "unused variable" and "unused local function"

warnings may be issued. It is our failure to do so which is

at the heart of the current issue, and as a porter of code, it

has been a recurrent problem.''

``There are also two capabilities which are missing, which have

been refered to as an IGNORABLE declaration and a

COMPILER:INVISIBLE-REFERENCES form, in BARMAR's message. I

actually think that both are needed. IGNORABLE states that it

doesn't matter if the user doesn't reference a variable.

COMPILER:INVISIBLE-REFERENCES (WITHOUT-REFERENCES ?) states that

the user is *expected* to reference a particular variable, and

that style warnings are appropriate if he does not.

``I think both situations are common in macros, and that because

these issues are so close to the core of the language semantics,

this should be regarded as something more than mere "style

warnings". I wouldn't require compilers and interpreters to

issue the warnings in any particular situation, but I do think

we can and should define the semantics of our variables sufficiently

to state when these warnings are and are not appropriate.

``I'd also like to see all of these extended to local functions

in the same manner as the DYNAMIC-EXTENT declaration.''

Pitman notes the following bug report, received by Symbolics. One very

important effect of solving this issue is that vendors will know what

to implement and users will know what to expect. Right now, vendors

are forced to implement their conscience, and they have no defense if

a user thinks they did the wrong thing.

Date: Fri, 14 Sep 90 17:45 EDT

From: J.P. Massar <massar@Think.COM>

Subject: Stupid warnings

To: bug-lispm@Think.COM

Look at all this crap. I don't care what anyone says about

how iteration variables REALLY aren't been used...

I JUST DON'T WANT TO SEE THESE STUPID MESSAGES AND I DON'T

PARTICULARLY FEEL LIKE GOING AND PUTTING STUPID #+ WHATEVER's

OVER THIS CODE.

SYMBOLICS SHOULD BE SHOT FOR ISSUING THIS WARNING.

...

The only little problem is--all Symbolics did was implement what it

felt CLtL said to implement. What the user is really complaining

about is that the implementation didn't do the same thing as he

expected, and the only way it ever will is if we resolve this issue

once and for all.


[Starting Points][Contents][Index][Symbols][Glossary][Issues]
Copyright 1996-2005, LispWorks Ltd. All rights reserved.