This is ../info/internals.info, produced by makeinfo version 4.8 from
internals/internals.texi.

INFO-DIR-SECTION XEmacs Editor
START-INFO-DIR-ENTRY
* Internals: (internals).       XEmacs Internals Manual.
END-INFO-DIR-ENTRY

   Edition History:

   Created November 1995 (?) by Ben Wing.  XEmacs Internals Manual
Version 1.0, March, 1996.  XEmacs Internals Manual Version 1.1, March,
1997.  XEmacs Internals Manual Version 1.4, March, 2001.  XEmacs
Internals Manual Version 21.5, October, 2004.

   Copyright (C) 1992 - 2004 Ben Wing.  Copyright (C) 1996, 1997 Sun
Microsystems.  Copyright (C) 1994 - 1998, 2002, 2003 Free Software
Foundation.  Copyright (C) 1994, 1995 Board of Trustees, University of
Illinois.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that the
entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the Foundation.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided also
that the section entitled "GNU General Public License" is included
exactly as in the original, and provided that the entire resulting
derived work is distributed under the terms of a permission notice
identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that the section entitled "GNU General Public License"
may be included in a translation approved by the Free Software
Foundation instead of in the original English.


File: internals.info,  Node: Future Work -- Specifiers,  Next: Future Work -- Making Elisp Function Calls Faster,  Prev: Future Work -- Macros,  Up: Future Work

44.19 Future Work - Specifiers
==============================

Author: Ben Wing (mailto:ben@xemacs.org)

Ideas To Work On When Their Time Has Come
-----------------------------------------

   * specifier-instance returns additional params (multiple-value) -
     the instantiator used, the associated tag set, the locale found
     in, a code that can be passed in as an additional param RESTART to
     restart an instantiation process, e.g. to allow an instantiator to
     "inherit" from another one higher up.  Also, domain can be 'global
     (look only in global specs) or "complex" - a list of the actual
     locales to look in (e.g. a buffer - frame - a device - 'global)

   * pragmatic-specifier-domain (locale) Converts a locale into a
     domain in a way that's "pragmatic" - does what most users expect
     will happen, but is not clean.  In particular, handling of
     "buffer" requires trickiness, as mentioned before.

   * ensure-instantiator-exists (specifier locale) Ensures an actual
     instantiator exists in a locale, so that it can later be futzed
     with.  If none exists, one is constructed by first calling
     pragmatic-specifier domain and then specifier-instance and
     fetching out the instantiator for this call.

   * map-modifying-instantiators (specifier fun &optional locale
     tag-set) Same args as map-specifier, but use the return value from
     the fun to replace the instantiator.  Called with three args
     (instantiator locale tag-set)

   * map-modifying-instantiators-force (specifier fun &optional locale
     tag-set) Same as previous, but calls ensure-instantiator-exists on
     each locale before processing.

   NOTE:  Can do preliminary implementation without Multiple Values -
instead create fun specifier-instance - that returns a list (and will
be deleted at some point)

specifier &c changes for glyphs
-------------------------------

  1.        * resizable vectors with funs to insert, delete elements
          (elements shift accordingly)

        * gap array vectors as an implementation of resizing vectors.

  2. You can `put' `get', etc. on vectors to modify properties within
     them.

  3. copy-over routines routines that carefully copy one complex item
     OVER another one, destroying the second in the process.  I wrote
     one for lists.  Need a general copy-over-tree.

  4. improvement to specifier mapping routines e.g.

     map-modifying-instantiator and its force versions below, so that we
     could implement in turns.

  5. put-specifier-property (specifier which finds the key, value
     instantiator in the locale, &opt locale possibly creating one
     tag-set) if necessary and goes into the vector, changes it, and
     puts it back into the specifier.

  6. Smarter add-spec-to-specifier

     If it notices that it's just replacing one instantiator with
     another, instead of just copy-tree the first one and throw away the
     other, use copy-over-tree to save lots of garbage when repeatedly
     called.

     ILLEGIBLE: GOTO LOO BUI BUGS LAST PNOTE

  7. When at image instantiate:
        * Some properties in the instantiators could be implemented
          through dynamically modifying an existing image instance
          (e.g. when the value of a slider or progress bar or text in a
          text field changes).  So when we hash, we only hash the part
          of the instantiator that cannot be dynamically modified (We
          might need to do something tricky here - allowing a :key
          property in hash tables or *ILLEGIBLE*).  Anyway, so we need
          to generate an image instance, and we mask off the dynamic
          properties and look up in our hash table, and we get
          something back!  But is it ours to modify?  (We already
          checked to see it wasn't exactly the same dynamic properties
          that it had)  Thus --

  8. Reference counting.  Somehow or other, each image instance in the
     cache needs to keep track of the instantiators that generated it.

   It might do this through some sort of special instantiator-reference
object.  This points to the instantiator, where in the hierarchy the
instantiator is etc.  When an instantiator gets removed, this
gu*ILLEGIBLE* values report not attached.  Somehow that gets
communicated back to the image instance in the cache.  So somehow or
other, the image instance in the cache knows who's using them and so
when you go and keep updating the slider value, by simply modifying an
instantiator, which efficiently changes the internal structure of this
specifier - eventually image instantiate notices that the image
instance it points has no other user and just modifiers it,  but in
complex situations, some optimizations get lost, but everything is
still correct.

   vs.

   Andy's set-image-instance-property, which achieves the same
optimizations much more easily, but

  1. falls apart in any more complicated system

  2. only works because of the way the caching system in XEmacs works.
     Any change (e.g. *ILLEGIBLE* more of making the caches GQ instead
     of GQ) is likely to make things stop working right in all but the
     simplest situation.

Specifier improvements for support of specifier inheritance (necessary for the new font mapping API)
----------------------------------------------------------------------------------------------------

'Fallback should be a locale/domain.

     (get-specifier specifier &optional locale)

     #### If locale is omitted, should it be (current-buffer) or 'global?
     #### Should argument not be optional?

   If a buffer is specified: find a window showing buffer by looking

   * at selected window

   * at other windows on selected frame

   * at selected windows on other frames in selected device

   * at other windows on ""

   * at selected windows on selected frames on other devices in selected
     console.

   * other windows sel from other devices sel con

   * ""       oth       ""           sel

   * sel win sel from sel dev oth con

   * oth win sel from sel dev oth con

   * sel win oth from sel dev oth con

   * oth win oth from sel dev oth con

   * sel win sel from oth dev oth con

   * oth win sel from oth dev oth con

   * oth win oth from oth dev oth con

   If none, use  buffer -> sel from -> etc.

     Returns multiple values
       second is instantiator
       third  is locale containing inst.
       fourth is tag set

     (restart-specifier-instance ...)

   like specifier-instance, but allows restarting the lookup, for
implementing inheritance, etc.  Obsoletes
specifier-matching-find-charset, or whatever it is.  The restart
argument is opaque, and is returned as a multiple value of
restart-specifier-instance.  (It's actually an integer with the low
bits holding the locale and the other bits count int to the list)
attached to the locale.)


File: internals.info,  Node: Future Work -- Making Elisp Function Calls Faster,  Next: Future Work -- Lisp Engine Replacement,  Prev: Future Work -- Specifiers,  Up: Future Work

44.20 Future Work - Making Elisp Function Calls Faster
======================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract: *This page describes many optimizations that can be made
to the existing Elisp function call mechanism without too much effort.
The most important optimizations can probably be implemented with only
a day or two of work.  I think it's important to do this work
regardless of whether we eventually decide to replace the Lisp engine.

   Many complaints have been made about the speed of Elisp, and in
particular about the slowness in executing function calls, and rightly
so.  If you look at the implementation of the `funcall' function,
you'll notice that it does an incredible amount of work.  Now logically,
it doesn't need to be so.  Let's look first from the theoretical
standpoint at what absolutely needs to be done to call a Lisp function.

   First, let's look at the situation that would exist if we were smart
enough to have made lexical scoping be the default language policy.  We
know at compile time exactly which code can reference the variables that
are the formal parameters for the function being called (specifically,
only the code that is part of that function's definition) and where
these references are.  As a result, we can simply push all the values of
the variables onto a stack, and convert all the variable references in
the function definition into stack references.  Therefore, binding
lexically-scoped parameters in preparation for a function call involves
nothing more than pushing the values of the parameters onto a stack and
then setting a new value for the frame pointer, at the same time
remembering the old one.  Because the byte-code interpreter has a
stack-based architecture, however, the parameter values have already
been pushed onto the stack at the time of the function call invocation.
Therefore, binding the variables involves doing nothing at all, other
than dealing with the frame pointer.

   With dynamic scoping, the situation is somewhat more complicated.
Because the parameters can be referenced anywhere, and these references
cannot be located at compile time, their values have to be stored into a
global table that maps the name of the parameter to its current value.
In Elisp, this table is called the "obarray".  Variable binding in
Elisp is done using the C function `specbind()'. (This stands for
"special variable binding" where "special" is the standard Lisp
terminology for a dynamically-scoped variable.)  What `specbind()'
does, essentially, is retrieve the old value of the variable out of the
obarray, remember the value by pushing it, along with the name of the
variable, onto what's called the "specpdl" stack, and then store the
new value into the obarray.  The term "specpdl" means "Special Variable
Pushdown List", where "Pushdown List" is an archaic computer science
term for a stack that used to be popular at MIT.  These binding
operations, however, should still not take very much time because of the
use of symbols, i.e. because the location in the obarray where the
variable's value is stored has already been determined (specifically, it
was determined at the time that the byte code was loaded and the symbol
created), so no expensive hash table lookups need to be performed.

   An actual function invocation in Elisp does a great deal more work,
however, than was just outlined above.  Let's just take a look at what
happens when one byte-compiled function invokes another byte-compiled
function, checking for places where unnecessary work is being done and
determining how to optimize these places.

  1.  The byte-compiled function's parameter list is stored in exactly
     the format that the programmer entered it in, which is to say as a
     Lisp list, complete with `&amp;optional' and `&amp;rest' keywords.
     This list has to be parsed for _every_ function invocation, which
     means that for every element in a list, the element is checked to
     see whether it's the `&amp;optional' or `&amp;rest' keywords, its
     surrounding cons cell is checked to make sure that it is indeed a
     cons cell, the `QUIT' macro is called, etc.  What should be
     happening here is that the argument list is parsed exactly once,
     at the time that the byte code is loaded, and converted into a C
     array.  The C array should be stored as part of the byte-code
     object.  The C array should also contain, in addition to the
     symbols themselves, the number of required and optional arguments.
     At function call time, the C array can be very quickly retrieved
     and processed.

  2.  For every variable that is to be bound, the `specbind()' function
     is called.  This actually does quite a lot of things, including:

       1.  Checking the symbol argument to the function to make sure
          it's actually a symbol.

       2.  Checking for specpdl stack overflow, and increasing its size
          as necessary.

       3.  Calling `symbol_value_buffer_local_info()' to retrieve
          buffer local information for the symbol, and then processing
          the return value from this function in a series of if
          statements.

       4.  Actually storing the old value onto the specpdl stack.

       5.  Calling `Fset()' to change the variable's value.



   The entire series of calls to `specbind()' should be inline and
merged into the argument processing code as a single tight loop, with no
function calls in the vast majority of cases.  The `specbind()' logic
should be streamlined as follows:

  1.  The symbol argument type checking is unnecessary.

  2.  The check for the specpdl stack overflow needs to be done only
     once, not once per argument.

  3.  All of the remaining logic should be boiled down as follows:

       1.  Retrieve the old value from the symbol's value cell.

       2.  If this value is a symbol-value-magic object, then call the
          real `specbind()' to do the work.

       3.  Otherwise, we know that nothing complicated needs to be
          done, so we simply push the symbol and its value onto the
          specpdl stack, and then replace the value in the symbol's
          value cell.

       4.  The only logic that we are omitting is the code in `Fset()'
          that checks to make sure a constant isn't being set.  These
          checks should be made at the time that the byte code for the
          function is loaded and the C array of parameters to the
          function is created.  (Whether a symbol is constant or not is
          generally known at XEmacs compile time.  The only issue here
          is with symbols whose names begin with a colon.  These
          symbols should simply be disallowed completely as parameter
          names.)



   Other optimizations that could be done are:

   *  At the beginning of the function that implements the byte-code
     interpreter (this is the Lisp primitive `byte-code'), the string
     containing the actual byte code is converted into an array of
     integers.  I added this code specifically for MULE so that the
     byte-code engine didn't have to deal with the complexities of the
     internal string format for text.  This conversion, however, is
     generally useful because on modern processors accessing 32-bit
     values out of an array is significantly faster than accessing
     unaligned 8-bit values.  This conversion takes time, though, and
     should be done once at load time rather than each time the byte
     code is executed.  This array should be stored in the byte-code
     object.  Currently, this is a bit tricky to do, because
     `byte-code' is not actually passed the byte-code object, but
     rather three of its elements.  We can't just change `byte-code' so
     that it is directly passed the byte-code object because this
     function, with its existing argument calling pattern, is called
     directly from compiled Elisp files.  What we can and should do,
     however, is create a subfunction that does take a byte-code object
     and actually implements the byte-code interpreter engine.
     Whenever the C code wants to execute byte code, it calls this
     subfunction.  `byte-code' itself also calls this subfunction after
     conjuring up an appropriate byte-code object and storing its
     arguments into this object.  With a small amount of work, it's
     possible to do this conjuring in such a way that it doesn't
     generate any garbage.

   *  At the end of a function call, the parameter bindings that have
     been done need to be undone.  This is standardly done by calling
     `unbind_to()'.  Just as for a `specbind()', this function does a
     lot of work that is unnecessary in the vast majority of cases, and
     it could also be inlined and streamlined.

   *  As part of each Elisp function call, a whole bunch of checks are
     done for a series of unlikely but possible conditions that may
     occur.  These include, for example,

        *  Calling the `QUIT' macro, which essentially involves
          checking a global volatile variable to see whether additional
          processing needs to be done.

        *  Checking whether a garbage collection needs to be done.

        *  Checking the variable `debug_on_next_call'.

        *  Checking for whether Elisp profiling is active.  (An
          additional optimization that's perhaps not worth the effort
          is to do some post-processing on the array of integers after
          it has been converted.  For example, whenever a 16-bit value
          occurs in the byte code, it has to be encoded as two separate
          8-bit values.  These values could be combined.  The tricky
          part here is that all of the places where a goto occurs
          across the place where this modification is made would have to
          have their offsets changed.  Other such optimizations can
          easily be imagined as well.)


   *  With a little bit smarter code, it should be possible to make a
     single trip variable that indicates whether any of these
     conditions is true.  This variable would be updated by any code
     that changes the actual variables whose values are checked in the
     various checks just mentioned.  (By the way, all of this is
     occurring in the C function `funcall_recording_as()'.)  There is a
     little bit of code between each of the checks.  This code would
     simply have to be duplicated between the two cases where this
     general trip variable is true and is false.  (Note: the
     optimization detailed in this item is probably not worth doing on
     the first pass.)



File: internals.info,  Node: Future Work -- Lisp Engine Replacement,  Next: Future Work -- Better Rendering Support,  Prev: Future Work -- Making Elisp Function Calls Faster,  Up: Future Work

44.21 Future Work - Lisp Engine Replacement
===========================================

* Menu:

* Future Work -- Lisp Engine Discussion::
* Future Work -- Lisp Engine Replacement -- Implementation::
* Future Work -- Startup File Modification by Packages::


File: internals.info,  Node: Future Work -- Lisp Engine Discussion,  Next: Future Work -- Lisp Engine Replacement -- Implementation,  Prev: Future Work -- Lisp Engine Replacement,  Up: Future Work -- Lisp Engine Replacement

44.21.1 Future Work - Lisp Engine Discussion
--------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract: *Recently there has been a great deal of talk on the
XEmacs mailing lists about potential changes to the XEmacs Lisp engine.
Usually the discussion has centered around the question which is better,
Common Lisp or Scheme?  This is certainly an interesting debate topic,
but it didn't seem to have much practical relevance to me, so I vowed to
stay out of the discussion.  Recently, however, it seems that people are
losing sight of the broader picture.  For example, nobody seems to be
asking the question, ""Would an extension language other than Lisp or
Scheme (perhaps not a Lisp variant at all) be more appropriate?"" Nor
does anybody seem to be addressing what I consider to be the most
fundamental question, is changing the extension language a good thing to
do?

   I think it would be a mistake at this point in XEmacs development to
begin any project involving fundamental changes to the Lisp engine or to
the XEmacs Lisp language itself.  It would take a huge amount of effort
to complete even part of this project, and would be a major drain on the
already-insufficient resources of the XEmacs development community.
Most of the gains that are purported to stem from a project such as this
could be obtained with far less effort by making more incremental
changes to the XEmacs core.  I think it would be an even bigger mistake
to change the actual XEmacs extension language (as opposed to just
changing the Lisp engine, making few, if any, externally visible
changes).  The only language change that I could possibly imagine
justifying would involve switching to some ubiquitous web language, such
as Java and JavaScript, or Perl.  (Even among those, I think Java would
be the only possibility that really makes sense).

   In the rest of this document I'll present the broader issues that
would be involved in changing the Lisp engine or extension language.
This should make clear why I've come to believe as I do.

Is everyone clear on the difference between interface and implementation?
-------------------------------------------------------------------------

There seems to be a great deal of confusion concerning the difference
between interface and implementation.  In the context of XEmacs,
changing the interface means switching to a different extension language
such as Common Lisp, Scheme, Java, etc.  Changing the implementation
means using a different Lisp engine.  There is obviously some relation
between these two issues, but there is no particular requirement that
one be changed if the other is changed.  It is quite possible, for
example, to imagine taking the underlying engine for any of the various
Lisp dialects in existence, and adapting it so that it implements the
same Elisp extension language that currently exists.  The vast majority
of the purported benefits that we would get from changing the extension
language could just as easily be obtained while making minimal changes
to the external Elisp interface.  This way nearly all existing Elisp
programs would continue to work, there would be no need to translate
Elisp programs into some other language or to simultaneously support two
incompatible Lisp variants, and there would be no need for users or
package authors to learn a new extension language that would be just as
unfamiliar to the vast majority of them as Elisp is.

Why should we change the Lisp engine?
-------------------------------------

Let's go over the possible reasons for changing the Lisp engine.

Speed.
......

Changing the Lisp engine might make XEmacs faster.  However, consider
the following.

  1.  XEmacs will get faster over time without any development effort
     at all because computers will get faster.

  2.  Perhaps the biggest causes of the slowness of XEmacs are not
     related to the Lisp engine at all.  It has been asserted, for
     example, that the slowness of XEmacs is primarily due to the
     redisplay mechanism, to the handling of insertion and deletion of
     text in a buffer, to the event loop, etc.  Nobody has done any
     real studies to determine what the actual cause of slowness is.

  3.  Emacs 18 seems plenty fast enough to most people.  However, Emacs
     18 also had a worse Lisp engine and a worse byte compiler than
     XEmacs.

  4.  Significant speed increases in the execution of Lisp code could be
     achieved without too much effort by working on the existing byte
     code interpreter and function call mechanism a bit.


Memory usage.
.............

A new Lisp engine with a better garbage collection mechanism might make
more efficient use of memory; for example, through the use of a
relocating garbage collector.  However, consider this:

  1.  A new Lisp engine would probably have a larger memory footprint,
     perhaps a significantly larger one.

  2.  The worst memory problems might not be due to Lisp object
     inefficiency at all.  The problems could simply be due mainly to
     the inefficient buffer representation.  Nobody has come up with
     any concrete numbers on where the real problem lies.


Robustness.
...........

A new Lisp engine might well be more robust.  (On the other hand, it
might not be.  It is not always easy to tell).  However, I think that
the biggest problems with robustness are in the part of the C code that
is not concerned with implementing the Lisp engine.  The redisplay
mechanism and the unexec mechanism are probably the biggest sources of
robustness problems.  I think the biggest robustness problems that are
related to the Lisp engine concern the use of GCPRO declarations.  The
entire GCPRO mechanism is ill-conceived and unsafe.  The only real way
to make this safe would be to do conservative garbage collection over
the C stack and to eliminate the GCPRO declarations entirely.  But how
many of the Lisp engines that are being considered have such a mechanism
built into them?

Maintainability.
................

A new Lisp engine might well improve the maintainability of XEmacs by
offloading the maintenance of the Lisp engine.  However, we need to make
very sure that this is, in fact, the case before embarking on a project
like this.  We would almost certainly have to make significant
modifications to any Lisp engine that we choose to integrate, and
without the active and committed support and cooperation of the
developers of that Lisp engine, the maintainability problem would
actually get worse.

Features.
.........

A new Lisp engine might have built in support for various features that
we would like to add to the XEmacs extension language, such as lexical
scoping and an object system.

Why would we want to change the extension language?
---------------------------------------------------

Possible reasons for changing the extension language include:

More standard.
..............

Switching to a language that is more standard and more commonly in use
would be beneficial for various reasons.  First of all, the language
that is more commonly used and more familiar would make it easier for
users to write their own extensions and in general, increase the
acceptance of XEmacs.  Also, an accepted standard probably has had a lot
more thought put into it than any language interface created by the
XEmacs developers themselves.  Furthermore, if our extension language is
being actively developed and supported, much of the work that we would
otherwise have to do ourselves is transferred elsewhere.

   However, both Scheme and Common Lisp flunk the familiarity test.
Neither language is being actively used for program development outside
of small research communities, and few prospective authors of XEmacs
extensions will be familiar with any Lisp variant for real world uses.
(I consider the argument that Scheme is often used in introductory
programming courses to be irrelevant.  Many existing programmers were
taught Pascal in their introductory programming courses.  How many of
them would actually be comfortable writing a program in Pascal?)
Furthermore, someone who wants to learn Lisp can't exactly go to their
neighborhood bookstore and pick up a book on this topic.

Ease of use.
............

There are endless arguments about which language is easiest to use.  In
practice, this largely boils down to which languages are most familiar.

Object oriented.
................

The object-oriented paradigm is the dominant one in use today for new
languages.  User interface concepts in particular are expressed very
naturally in an object-oriented system.  However, neither Scheme nor
Common Lisp has been designed with object orientation in mind.  There is
a standard object system for Common Lisp, but it is extremely complex
and difficult to understand.


File: internals.info,  Node: Future Work -- Lisp Engine Replacement -- Implementation,  Next: Future Work -- Startup File Modification by Packages,  Prev: Future Work -- Lisp Engine Discussion,  Up: Future Work -- Lisp Engine Replacement

44.21.2 Future Work - Lisp Engine Replacement - Implementation
--------------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Let's take a look at the sort of work that would be required if we
were to replace the existing Elisp engine in XEmacs with some other
engine, for example, the Clisp engine.  I'm assuming here, of course,
that we are not going to be changing the interface here at the same
time, which is to say that we will be keeping the same Elisp language
that we currently have as the extension language for XEmacs, except
perhaps for incremental changes that we will make, such as lexical
scoping and proper structure support in an attempt to gradually move
the language towards an upwardly-compatible goal, such as Common Lisp.
I am writing this page primarily as food for thought.  I feel fairly
strongly that actually doing this work would be a big waste of effort
that would inevitably become a huge time sink on the part of nearly
everyone involved in XEmacs development, and not only for the ones who
were supposed to be actually doing the engine change.  I feel that most
of the desired changes that we want for the language and/or the engine
can be achieved with much less effort and time through incremental
changes to the existing code base.

   First of all, in order to make a successful Lisp engine change in
XEmacs, it is vitally important that the work be done through a series
of incremental stages where at the end of each stage XEmacs can be
compiled and run, and it works.  It is tempting to try to make the
change all at once, but this would be disastrous.  If the resulting
product worked at all, it would inevitably contain a huge number of
subtle and extremely difficult to track down bugs, and it would be next
to impossible to determine which of the myriad changes made introduced
the bug.

   Now let's look at what the possible stages of implementation could
be.

An Extra C Preprocessing Stage
..............................

The first step would be to introduce another preprocessing stage for the
XEmacs C code, which is done before the C compiler itself is invoked on
the code, and before the standard C preprocessor runs.  The C
preprocessor is simply not powerful enough to do many of the things we
would like to do in the C code.  The existing results of this have been
a combination of a lot of hacked up and tricky-to-maintain stuff (such
as the `DEFUN' macro, and the associated `DEFSUBR'), as well as code
constructs that are difficult to write.  (Consider for example,
attempting to do structured exception handling, such as catch/throw and
unwind-protect constructs), as well as code that is potentially or
actually unsafe (such as the uses of `alloca'), which could easily
cause stack overflow with large amounts of memory allocated in this
fashion.)  The problem is that the C preprocessor does not allow macros
to have the power of an actual language, such as C or Lisp.  What our
own preprocessor should do is allow us to define macros, whose
definitions are simply functions written in some language which are
executed at compile time, and whose arguments are the actual argument
for the macro call, as well as an environment which should have a data
structure representation of the C code in the file and allow this
environment to be queried and modified.  It can be debated what the
language should be that these extensions are written in.  Whatever the
language chosen, it needs to be a very standard language and a language
whose compiler or interpreter is available on all of the platforms that
we could ever possibly consider putting XEmacs to, which is basically to
say all the platforms in existence.  One obvious choice is C, because
there will obviously be a C compiler available, because it is needed to
compile XEmacs itself.  Another possibility is Perl, which is already
installed on most systems, and is universally available on all others.
This language has powerful text processing facilities which would
probably make it possible to implement the macro definitions more
quickly and easily; however, this might also encourage bad coding
practices in the macros (often simple text processing is not
appropriate, and more sophisticated parsing or recursive data structure
processing needs to be done instead), and we'd have to make sure that
the nested data structure that comprises the environment could be
represented well in Perl.  Elisp would not be a good choice because it
would create a bootstrapping problem.  Other possible languages, such as
Python, are not appropriate, because most programmers are unfamiliar
with this language (creating a maintainability problem) and the Python
interpreter would have to be included and compiled as part of the XEmacs
compilation process (another maintainability problem).  Java is still
too much in flux to be considered at this point.

   The macro facility that we will provide needs to add two features to
the language: the ability to define a macro, and the ability to call a
macro.  One good way of doing this would be to make use of special
characters that have no meaning in the C language (or in C++ for that
matter), and thus can never appear in a C file outside of comments and
strings.  Two obvious characters are the @ sign and the $ sign.  We
could, for example, use `@' defined to define new macros, and the `$'
sign followed by the macro name to call a macro.  (Proponents of Perl
will note that both of these characters have a meaning in Perl.  This
should not be a problem, however, because the way that macros are
defined and called inside of another macro should not be through the use
of any special characters which would in effect be extending the macro
language, but through function calls made in the normal way for the
language.)

   The program that actually implements this extra preprocessing stage
needs to know a certain amount about how to parse C code.  In
particular, it needs to know how to recognize comments, strings,
character constants, and perhaps certain other kinds of C tokens, and
needs to be able to parse C code down to the statement level.  (This is
to say it needs to be able to parse function definitions and to separate
out the statements, `if' blocks, `while' blocks, etc. within these
definitions.  It probably doesn't, however need to parse the contents
of a C expression.)  The preprocessing program should work first by
parsing the entire file into a data structure (which may just contain
expressions in the form of literal strings rather than a data structure
representing the parsed expression).  This data structure should become
the environment parameter that is passed as an argument to macros as
mentioned above.  The implementation of the parsing could and probably
should be done using `lex' and `yacc'.  One good idea is simply to
steal some of the `lex' and `yacc' code that is part of GCC.

   Here are some possibilities that could be implemented as part of the
preprocessing:

  1.  A proper way of doing the `DEFUN' macros.  These could, for
     example, take an argument list in the form of a Lisp argument list
     (complete with keyword parameters and other complex features) and
     automatically generate the appropriate `subr' structure, the
     appropriate C function definition header, and the appropriate call
     to the `DEFSUBR' initialization function.

  2.  A truly safe and easy to use implementation of the `alloca'
     function.  This could allocate the memory in any fashion it chooses
     (calling `malloc' using a large global array, or a series of such
     arrays, etc.) an `insert' in the appropriate places to
     automatically free up this memory.  (Appropriate places here would
     be at the end of the function and before any return statements.
     Non-local exits can be handled in the function that actually
     implements the non-local exit.)

  3.  If we allow for the possibility of having an arbitrary Lisp
     engine, we can't necessarily assume that we can call Lisp
     primitives implemented in C from other C functions by simply
     making a function all.  Perhaps something special needs to happen
     when this is done.  This could be handled fairly easily by having
     our new and improved `DEFUN' macro define a new macro for use when
     calling a primitive.

Make the Existing Lisp Engine be Self-contained.
................................................

The goal of this stage is to gradually build up a self-contained Lisp
engine out of the existing XEmacs core, which has no dependencies on any
of the code elsewhere in the XEmacs core, and has a well-defined and
black box-style interface.  (This is to say that the rest of the C code
should not be able to access the implementation of the Lisp engine, and
should make as few assumptions as possible about how this implementation
works).  The Lisp engine could, and probably should, be built up as a
separate library which can be compiled on its own without any of the
rest of the XEmacs C code, and can be tested in this configuration as
well.

   The creation of this engine library should be done as a series of
subsets, each of which moves more code out of the XEmacs core and into
the engine library, and XEmacs should be compilable and runnable between
each sub-step.  One possible series of sub-steps would be to first
create an engine that does only object allocation and garbage
collection, then as a second sub-step, move in the code that handles
symbols, symbol values, and simple binding, and then finally move in the
code that handles control structures, function calling, `byte-code'
execution, exception handling, etc.  (It might well be possible to
further separate this last sub-step).

Removal of Assumptions About the Lisp Engine Implementation
...........................................................

Currently, the XEmacs C code makes all sorts of assumptions about the
implementation of the Lisp engine, particularly in the areas of object
allocation, object representation, and garbage collection.  A different
Lisp engine may well have different ways of doing these implementations,
and thus the XEmacs C code must be rid of any assumptions about these
implementations.  This is a tough and tedious job, but it needs to be
done.  Here are some examples:

  1.  `GCPRO' must go.  The `GCPRO' mechanism is tedious, error-prone,
     unmaintainable, and fundamentally unsafe.  As anyone who has
     worked on the C Core of XEmacs knows, figuring out where to insert
     the `GCPRO' calls is an exercise in black magic, and debugging
     crashes as a result of incorrect `GCPROing' is an absolute
     nightmare.  Furthermore, the entire mechanism is fundamentally
     unsafe.  Even if we were to use the extra preprocessing stage
     detailed above to automatically generate `GCPRO' and `UNGCPRO'
     calls for all Lisp object variables occurring anywhere in the C
     code, there are still places where we could be bitten.  Consider,
     for example, code which calls `cons' and where the two arguments
     to this functions are both calls to the `append' function.  Now
     the `append' function generates new Lisp objects, and it also
     calls `QUIT', which could potentially execute arbitrary Lisp code
     and cause a garbage collection before returning control to the
     `append' function.  Now in order to generate the arguments to the
     `cons' function, the `append' function is called twice in a row.
     When the first `append' call returns, new Lisp data has been
     created, but has no `GCPRO' pointers to it.  If the second
     `append' call causes a garbage collection, the Lisp data from the
     first `append' call will be collected and recycled, which is
     likely to lead to obscure and impossible-to-debug crashes.  The
     only way around this would be to rewrite all function calls whose
     parameters are Lisp objects in terms of temporary variables, so
     that no such function calls ever contain other function calls as
     arguments.  This would not only be annoying to implement, even in
     a smart preprocessor, but would make the C code become incredibly
     slow because of all the constant updating of the `GCPRO' lists.

  2.  The only proper solution here is to completely do away with the
     `GCPRO' mechanism and simply do conservative garbage collection
     over the C stack.  There are already portable implementations of
     conservative pointer marking over the C stack, and these could
     easily be adapted for use in the Elisp garbage collector.  If, as
     outlined above, we use an extra preprocessing stage to create a
     new version of `alloca' that allocates its memory elsewhere than
     actually on the C stack, and we ensure that we don't declare any
     large arrays as local variables, but instead use `alloca', then we
     can be guaranteed that the C stack is small and thus that the
     conservative pointer marking stage will be fast and not very
     likely to find false matches.

  3.  Removing the `GCPRO' declarations as just outlined would also
     remove the assumption currently made that garbage collection can
     occur only in certain places in the C code, rather than in any
     arbitrary spot.  (For example, any time an allocation of Lisp data
     happens).  In order to make things really safe, however, we also
     have to remove another assumption as detailed in the following
     item.

  4.  Lisp objects might be relocatable.  Currently, the C code assumes
     that Lisp objects other than string data are not relocatable and
     therefore it's safe to pass around and hold onto the actual
     pointers for the C structures that implement the Lisp objects.
     Current code, for example, assumes that a `Lisp_Object' of type
     buffer and a C pointer to a `struct buffer' mean basically the
     same thing, and indiscriminately passes the two kinds of buffer
     pointers around.  With relocatable Lisp objects, the pointers to
     the C structures might change at any time.  (Remember, we are now
     assuming that a garbage collection can happen at basically any
     point).  All of the C code needs to be changed so that Lisp
     objects are always passed around using a Lisp object type, and the
     underlying pointers are only retrieved at the time when a
     particular data element out of the structure is needed.  (As an
     aside, here's another reason why Lisp objects, instead of
     pointers, should always be passed around.  If pointers are passed
     around, it's conceivable that at the time a garbage collection
     occurs, the only reference to a Lisp object (for example, a
     deleted buffer) would be in the form of a C pointer rather than a
     Lisp object.  In such a case, the conservative pointer marking
     mechanism might not notice the reference, especially if, in an
     attempt to eliminate false matches and make the code generally
     more efficient, it will be written so that it will look for actual
     Lisp object references.)

  5.  I would go a step farther and completely eliminate the macros that
     convert a Lisp object reference into a C pointer.  This way the
     only way to access an element out of a Lisp object would be to use
     the macro for that element, which in one atomic operation
     de-references the Lisp object reference and retrieves the value
     contained in the element.  We probably do need the ability to
     retrieve actual C pointers, though.  For example, in the case
     where an array is stored in a Lisp object, or simply for
     efficiency purposes where we might want some code to retrieve the
     C pointer for a Lisp object, and work on that directly to avoid a
     whole bunch of extra indirections.  I think the way to do this
     would be through the use of a special locking construct
     implemented as part of the extra preprocessor stage mentioned
     above.  This would essentially be what you might call a "lock
     block", just like a `while' block.  You'd write the word `lock'
     followed by a parenthesized expression that retrieves the C
     pointer and stores it into a variable that is scoped only within
     the lock block and followed in turn by some code in braces, which
     is the actual code associated with the lock block, and which can
     make use of this pointer.  While the code inside the lock block is
     executing, that particular pointer and the object pointed to by it
     is guaranteed not to be relocated.

  6.  If all the XEmacs C code were converted according to these rules,
     there would be no restrictions on the sorts of implementations
     that can be used for the garbage collector.  It would be possible,
     for example, to have an incremental asynchronous relocating
     garbage collector that operated continuously in another thread
     while XEmacs was running.

  7.  The C implementation of Lisp objects might not, and probably
     should not, be visible to the rest of the XEmacs C code.  It
     should theoretically be possible, for example, to implement Lisp
     objects entirely in terms of association lists, rather than using
     C structures in the standard way.  (This may be an extreme
     example, but it's good to keep in mind an example such as this
     when cleaning up the XEmacs C code).  The changes mentioned in the
     previous item would go a long way towards removing this
     assumption.  The only places where this assumption might still be
     made would be inside of the lock blocks where an actual pointer is
     retrieved.  (Also, of course, we'd have to change the way that
     Lisp objects are defined in C so that this is done with some
     function calls and new and improved macros rather than by having
     the XEmacs C code actually define the structures.  This sort of
     thing would probably have to be done in any case once the
     allocation mechanism is moved into a separate library.)  With some
     thought it should be possible to define the lock block interface
     in such a way as to remove any assumptions about the
     implementation of Lisp objects.

  8.  C code may not be able to call Lisp primitives that are defined
     in C simply by making standard C function calls.  There might need
     to be some wrapper around all such calls.  This could be achieved
     cleanly through the extra preprocessing step mentioned above, in
     line with the example described there.


Actually Replacing the Engine.
..............................

Once we've done all of the work mentioned in the previous steps (and
admittedly, this is quite a lot of work), we should have an XEmacs that
still uses what is essentially the old and previously existing Lisp
engine, but which is ready to have its Lisp engine replaced.  The
replacement might proceed as follows:

  1.  Identify any further changes that need to be made to the engine
     interface that we have defined as a result of the previous steps
     so that features and idiosyncrasies of various Lisp engines that
     we examine could be properly supported.

  2.  Pick a Lisp engine and write an interface layer that sits on top
     of this Lisp engine and makes it adhere to what I'll now call the
     XEmacs Lisp engine interface.

  3.  Strongly consider creating, if we haven't already done so, a test
     suite that can test the XEmacs Lisp engine interface when used
     with a stand-alone Lisp engine.

  4.  Test the hell out of the Lisp engine that we've chosen when
     combined with its XEmacs Lisp engine interface layer as a
     stand-alone program.

  5.  Now finally attach this stand-alone program to XEmacs itself.
     Debug and fix any further problems that ensue (and there
     inevitably will be such problems), updating the test suite as we
     go along so that if it were run again on the old and buggy
     interfaced Lisp engine, it would note the bug.



File: internals.info,  Node: Future Work -- Startup File Modification by Packages,  Prev: Future Work -- Lisp Engine Replacement -- Implementation,  Up: Future Work -- Lisp Engine Replacement

44.21.3 Future Work - Startup File Modification by Packages
-----------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   OK, we need to create a design document for all of this, including:

   PRINCIPLE #1: Whenever you have auto-generated stuff, *CLEARLY*
indicate this in comments around the stuff.  These comments get
searched for, and used to locate the existing generated stuff to
replace.  Custom currently doesn't do this.

   PRINCIPLE #2: Currently, lots of functions want to add code to the
`init.el'. (e.g. I get prompted for my mail address from
add-change-log-entry, and then prompted if I want to make this
permanent).  There needs to be a Lisp API for working with arbitrary
code to be added to a user's startup.  This API hides all the details
of which file to put the fragment in, where in it, how to mark it with
magical comments of the right kind so that previous fragments can be
replaced, etc.

   PRINCIPLE #3: *ALL* generated stuff should be loaded before any
user-written init stuff.  This way the user can override the generated
settings.  Although in the case of customize, it may work when the
custom stuff is at the end of the init file, it surely won't work for
arbitrary code fragments (which typically do `setq' or the like).

   PRINCIPLE #4: As much as possible, generated stuff should be place in
separate files from non-generated stuff.  Otherwise it's inevitable
that some corruption is going to result.

   PRINCIPLE #5: Packages are encouraged, as much as possible, to work
within the customize model and store all their customizations there.
However, if they really need to have their own init files, these files
should be placed in .xemacs/, given normal names (e.g.
`saved-abbrevs.el' not .abbrevs), and there should be some magic
comment at the top of the file that causes it to get automatically
loaded while loading a user's init file. (Alternatively, the
above-named API could specify a function that lets a package specify
that they want such-and-such file loaded from the init file, and have
the specifics of this get handled correctly.)

   OVERARCHING GOAL: The overarching goal is to provide a unified
mechanism for packages to store state and setting information about the
user and what they were doing when XEmacs exited, so that the same or a
similar environment can be automatically set up the next time.  In
general, we are working more and more towards being a truly GUI app
where users' settings are easy to change and get remembered correctly
and consistently from one session to the next, rather than requiring
nasty hacking in elisp.

   Hrvoje, do you have any interest in this?  How about you, Martin?
This seems like it might be up your alley.  This stuff has been
ad-hocked since kingdom come, and it's high time that we make this work
properly so that it could be relied upon, and a lot of things could
"just work".


File: internals.info,  Node: Future Work -- Better Rendering Support,  Prev: Future Work -- Lisp Engine Replacement,  Up: Future Work

44.22 Future Work - Better Rendering Support
============================================

This section was written by Stephen Turnbull <stephen@xemacs.org>, so
don't blame Ben (or Eric and Matthias, for that matter).  Feel free to
add, edit, and share the blame, guys!

   As of late November 2004, this principally means adding support for
the `Xft' library, which provides a more robust _font configuration_
mechanism via Keith Packard's `fontconfig' library improved glyph
rendering, including antialiasing, via the `freetype' library, and
client-side rendering (saving bandwidth and server memory) via the
`XRender extension'.  In fact, patches which provide Xft support have
been available for several years, but the authors have been unwilling
to deal with several important issues which block integration.  These
are _Mule_, and more generally, _face_ support; _widget_ support
(including the toolbar and menubar); and _redisplay refactoring_.

   However, in late 2003 Eric Knauel
<knauel@informatik.uni-tuebingen.de> and Matthias Neubauer
<neubauer@informatik.uni-freiburg.de> put forward a relatively complete
patch which was robust to daily use in ISO 8859-1 locales, and Stephen
Turnbull began work on the integration issues.  At this point a
(private) CVS branch is available for Stephen's patch (branch point tag
`sjt-xft-bp', branch tag `sjt-xft'), and one may be made available for
the Knauel-Matthias patch soon.

* Menu:

* Better Rendering Support -- Review Criteria::
* Better Rendering Support -- Implementation::
* Better Rendering Support -- Current Status::
* Better Rendering Support -- Configuration with the Interim Patches::
* Better Rendering Support -- Modern Font Support::


File: internals.info,  Node: Better Rendering Support -- Review Criteria,  Next: Better Rendering Support -- Implementation,  Prev: Future Work -- Better Rendering Support,  Up: Future Work -- Better Rendering Support

44.22.1 Better Rendering Support - Review Criteria
--------------------------------------------------

Of course it's "unfair" to demand that the implementers of a nice
feature like anti-aliasing support deal with accumulated cruft of the
last few years, but somebody must, sometime soon.  Even core developers
are complaining about how slow XEmacs is in some applications, and there
is reason to believe that some of the problem is in redisplay.  Adding
more _ad hoc_ features to redisplay will make the whole module more
complex and unintelligible.  Even if it doesn't inherently further
detract from efficiency, it will surely make reform and refactoring
harder.

   Similar considerations apply to Mule support.  If Xft support is not
carefully designed, or implemented with Mule support soon, it will
undoubtedly make later Mule implementation far more difficult than it
needs to be, and require redundant work be done (_e.g._, on `Options'
menu support).

   Besides the design issue--and many users are requesting more
flexibility, primarily face support, from the widgets--with widget
support there is also an aesthetic issue.  It is horribly unimpressive
to have clunky bitmapped fonts on the decorations when pleasant
antialiased fonts are available in the buffer.

   Finally, these issues interact.  Widgets and faces are inherently
heavyweight objects, requiring orders of magnitude more computation than
simply displaying a string in a fixed font.  This will have an
efficiency impact, of course.  And they interact with each other; Mule
was designed for use in buffers and display in Emacs windows--but a
widget's content is usually not a buffer, and widgets need not be
displayed in a window, but may appear in other contexts, especially in
the gutters.  So specifiers will probably have to be reworked, in order
to properly support display of different faces in non-buffer, non-window
contexts.


File: internals.info,  Node: Better Rendering Support -- Implementation,  Next: Better Rendering Support -- Current Status,  Prev: Better Rendering Support -- Review Criteria,  Up: Future Work -- Better Rendering Support

44.22.2 Better Rendering Support - Implementation
-------------------------------------------------

Stephen is thinking in terms of the following components of a
comprehensive proposal.

*Font configuration*
     In XEmacs, font configuration is handled via _faces_.  Currently
     XEmacs uses a special type of _font specifier_ to map XEmacs
     locales to font names.  Especially under X11, this can cause
     annoying problems because of the unreliability of X servers'
     mappings from `XLFD' names to X11 fonts, over which XEmacs has no
     influence whatsoever.  However, the `fontconfig' library which is
     used with `Xft' provides much more reliable mapping, along with a
     more reliably parsable naming scheme similar to that used by
     TrueType fonts on MS Windows and the Macintosh.  Since the
     capabilities of font specifiers and `fontconfig' overlap, we
     should consider using `fontconfig' instead of `XLFD' names.  This
     implies that use of `Xft''s rendering functionality should be
     separated from use of `fontconfig'.

*fontconfig*
     Fontconfig is dramatically different from the X model in several
     ways.  In particular, for the convenient interface _fontconfig
     always returns a font_.  However, the font returned need not be
     anything like the desired font.  This means that XEmacs must adopt
     a strategy of delegating the search to fontconfig, then
     sanity-checking the return, rather than trying to use the
     fontconfig API to search using techniques appropriate for the X11
     core font API.  (This isn't actually true.  fontconfig has more
     complex interfaces which allow listing a subset of fonts that
     match a pattern, and don't go out of their may to return something
     no matter what.  But the original patches didn't use this
     approach.)

*Font menus*
     The `Options->Font' and `Options->Font Sizes' menus are broken, by
     design, not just by `Xft'.  Although they work better in Eric and
     Matthias's patch than in Stephen's, even their version has the
     problem that many fonts are unavailable because they don't match
     the current size--which is very strange, since `Xft' fonts are of
     course scalable.  But the whole idea of requiring that the font
     match the size is strange.  And the `Options->Font Weights' menu
     is just disabled, and has been for eons.

*X resources*
     Currently in Stephen's patch there are _five_ treatments of font
     resources.  There are the `XEmacs.FACE.attributeFont' resources
     used to set a single global font specification.  In the widgets,
     some (still) have a `font' resource using the automatic `Xt'
     resource conversion to `FontStruct', some have separate `font' and
     `fcFontName' resources with the former automatically converted to
     `FontStruct' by `Xt' and the latter left as a string, to be
     converted by `FcParseName' later, and some have a single `font'
     resource which is converted to `FontStruct' by `Xt' or the latter
     left as a string, depending on whether `Xft' was enabled by
     `configure' or not.  There is also the `xftFont' resource which
     may be retargeted to use an Xt converter function, but currently
     simply just an alias for the `fcFontName' resource.

     Stephen thinks that all of these should be converted to use the
     face approach, perhaps with some way to set specifications for
     individual widgets, frames, or buffers.  This will require some
     careful design work to incorporate face support in the widgets.
     We should just accept any or all of `font', `fontSet', and
     `fontList' resources, treat them all as lists of font names,
     either `XLFD'- or `fontconfig'-style, parse them ourselves (ie,
     not use the `Xt' resource manager), and add them to font
     specifiers as appropriate.  But this will require a bit of thought
     to obey POLA vis-a-vis usual `Xt' conventions.

*Rendering engine objects*
     With the introduction of the "Xft patch," the X11, Macintosh, and
     MS Windows platforms are all able to support multiple font
     rendering engines in the same binary.  Generically, there are
     several tasks that must be accomplished to render text on the
     display.  In both cases the code is rather disorganized, with
     substantial cross-platform duplication of similar routines.  While
     it may not be worthwhile to go the whole way to
     `RENDERER_HAS_METHOD' and `MAYBE_RENDMETH', refactoring these
     modules around the notion of interfacing a "generic rendering
     engine interface" to "text" seems like a plausible way to focus
     this work.

     Further evidence for this kind of approach is a bug recently fixed
     in the `xft-sjt' branch.  XEmacs was crashing because the Athena
     Label widget tried to access a nonexistent font in its
     initialization routine.  The font didn't exist because although no
     core X11 font corresponding to the spec existed, an Xft font was
     found.  So the XEmacs font instance existed but it did not specify
     an X11 core font, only the Xft font.  When this object was used to
     initialize the font for the Label widget, None (0) was passed to
     XtSetArgs, then XtCreateWidget was called, and the internal
     initialization routine attempted to access that (nonexistent) font
     while computing an X11 graphics context (GC).

     A similar issue applies to colors, but there Xft colors keep the
     pixel data internally, so (serendipitously) the X11 color (_i.e._,
     pixel) member does get updated.

*Colors, fonts, and faces*
     Besides the rendering engine itself, the XEmacs implementations of
     these objects are poorly supported by current widget
     implementations, including the traditional menubar and toolbar, as
     well as the more recent button, tab control, and progress bar
     widgets.  The refactoring suggested under "Rendering engine
     objects" should be conducted with an eye to making these widgets
     support faces, perhaps even to the extent of allowing rendering to
     X pixmaps (which some Athena widgets support, although they will
     not support rendering via Xft directly). Especially with `XRender'
     technology this should not be horribly inefficient.

*Specifiers, charsets, and languages*
     Traditionally Mule uses a rather rigid and low-level abstraction,
     the _charset_, to characterize font repertoires.  Unfortunately,
     support for a given charset is generally neither necessary nor
     sufficient to support a language.  Worse, although X11's only
     means for indicating font repertoires is the font's _registry_,
     the actual repertoire of many fonts is either deficient or
     font-dependent.  The only convenience is that the registry maps
     directly to a Mule charset in most cases, and vice versa.

     To date, XEmacs Mule has supported identification of appropriate
     fonts to support a language's repertoire of characters by
     identifying the repertoire as a subset of a union of charsets.  To
     each charset there is a regular expression matching the registry
     portion of a font name.  Then instantiation of a font proceeds by
     identifying the specifier domain, and then walking down the list
     of specifications, matching the regexp against font names until a
     match is found.  That font is requested from the system, and if
     not found, the process continues similarly until a font that can
     be loaded is found.

     This has several problems.  First, there's no guarantee that the
     union will be disjoint.  This problem manifests both in the case
     of display of Unicode representations of text in the `POSIX'
     default locale, where glyphs are typically drawn from several
     inappropriate fonts.  A similar problem often occurs, though for a
     different reason, in multilingual messages composed using `Gnus''s
     `message-mode' and MIME support.  This problem _cannot_ be avoided
     with the current design; it is quite possible that a font desired
     in one context will be shadowed by a font intended to get higher
     priority in a semantically different but syntactically similar (as
     far as Mule can tell) context.  (Of course, one could attach a
     different face as a text property, but that requires programming
     support; it can't be done by user configuration.)  The problem is
     only exacerbated as more and more Unicode fonts, supporting large
     repertoires with substantial overlap across fonts, are designed
     and published.

     A second problem is that registry names are often inaccurate.  For
     example, the Japanese JIS X 0208 standard was first published in
     1978 (as a relabelling of an older standard).  It was then revised
     in 1983, again in 1990, and once again in 2000, with slight
     changes to the repertoire and mapping in each revision.
     Technically, these standards can be distinguished in properly
     named fonts as `jisx0208.1978', `jisx0208.1983', `jisx0208.1990',
     `jisx0208.2000', but all of them are commonly simply labelled
     `jisx0208', and Western distributors, of course, generally lack
     the expertise to correctly relabel them.

     A third problem is that you generally can't tell if there are
     "holes" in the repertoire until you try to display the glyph.

     All of this tends to break standard idioms for handling Mule fonts
     in `init' files because they depend on charsets being disjoint
     repertoires.

     The TrueType fonts (and the later OpenType standard) provides for a
     proper character set query (as a Boolean vector indexed by Unicode
     code points), as well as providing a list of supported languages.

     I propose that we take advantage of these latter facilities by
     allowing a font to be specified either as a string (a font name),
     or as a list whose head is the font name and whose tail is a list
     of languages and Mule charsets (for backward compatibility) that
     user intends to use the font to display.  This will probably
     require a change to the specifier code.

     As mentioned above, specifiers will probably also have to be
     enhanced to recognize `widget' locales and domains, instead of the
     current hack where special `widget' and `gui-element' faces are
     created.

*Customize*
     Customize needs to deal with all this stuff!!


File: internals.info,  Node: Better Rendering Support -- Current Status,  Next: Better Rendering Support -- Configuration with the Interim Patches,  Prev: Better Rendering Support -- Implementation,  Up: Future Work -- Better Rendering Support

44.22.3 Better Rendering Support - Current Status
-------------------------------------------------

Stephen has a branch containing his stuff in XEmacs CVS.  The branch
point tag is `sjt-xft-bp', roughly corresponding to XEmacs 21.5.18, and
branch tag is `sjt-xft'.

44.22.3.1 Bugs Reported in sjt-xft
..................................

`ChangeLogs'
     A lot of these, especially for Eric and Matthias's work, are
     missing.  Mea culpa.

`Options->Font'
`Options->Font Size'
     These menus don't work.  All fonts are greyed out.  All sizes are
     available, but many (most?) faces don't change size, in particular,
     `default' does not.

`Antialiased text bleeding outside of reported extent'
     On my PowerBook G4 Titanium 15" screen, X.org server v6.8.1,
     dimensions:    1280x833 pixels (433x282 millimeters),
     resolution:    75x75 dots per inch,   depth of root window:    24
     planes (yes, those dimensions are broken), with font "Bitstream
     Vera Sans Mono-16:dpi=75" antialiased text may bleed out of the
     extent reported by XftTextExtents and other such facilities.  This
     is most obvious with the underscore character in that font.  The
     bottom of the underscore is antialiased, and insertions or
     deletions in the same line before the underscore leave a series of
     "phantom" underlines.  Except that it doesn't happen on the very
     first such insertion or deletion after a window refresh.  A
     similar effect sometimes occurs with deletions at the end of the
     line (no, I can't define "sometimes").  See also comments in
     `redisplay-x.c', functions `x_output_string' and
     `x_output_display_block'.  (Mostly duplicated here.)

     I think this is probably an Xft bug, but I'm not sure.


File: internals.info,  Node: Better Rendering Support -- Configuration with the Interim Patches,  Next: Better Rendering Support -- Modern Font Support,  Prev: Better Rendering Support -- Current Status,  Up: Future Work -- Better Rendering Support

44.22.4 Better Rendering Support - Configuration with the Interim Patches
-------------------------------------------------------------------------

For Stephen's `sjt-xft' branch, you should keep the following in mind
when configuring:

   * Although the menus don't work, it is possible to specify fonts for
     _faces_ using `set-face-font' (and other specifier-changing
     functions).

     There currently is no explicit way to specify that a particular
     font be used only for a given language.  However, since many fonts
     support only a limited repertoire such as ISO 8859/1, you can use
     the precedence of specifications for a given specifier locale to
     get something of this effect for non-Latin character sets.  This
     will normally work rather poorly for multiple Latin character
     sets, however, because the repertoires tend to have large amounts
     of overlap.  Support for specifying font by _language_ as well as
     by character set is planned.

     Because fonts supporting other languages tend to support English as
     well, if you want to use one font for English and another for the
     other language, you must use the `append' method when adding font
     specifications for the _other_ language.

     However, this leaves you with a problem if you want to change the
     other language's font: you have to remove the existing
     specification so it won't shadow the new one when you append.

     I use `define-specifier-tag' like this:

          (define-specifier-tag 'lang-ja)
          ;; No, I don't try to do real work with this font!  But it makes it
          ;; obvious that I got the requested font.  :-)
          (set-face-font 'default "AirCut-14")
          (set-face-font 'default "Kochi Mincho-14" nil '(lang-ja) 'append)
          ;; Oops, too sober.  Try something to match AirCut.
          (set-face-font 'default "Mikachan-14"
                         nil '(lang-ja) 'remove-tag-set-append)

   * The only way to configure widget fonts at the present time is to
     use X resources (or hack the source and rebuild).  Currently
     supported widgets are
        * menubars

        * tab controls

     Here are the resources I use.  *Warning:* _This interface will
     change._  The tab control and menubar have separate Font and
     XftFont resources, and use the X resource manager to instantiate a
     FontStruct from the Font resource.  There is no converter facility
     for XftFont yet, and creating one that handles both FontStruct and
     XftFont depending on XEmacs's configuration and the font name
     seems error-prone at best.  Probably we will should to a simple
     string representation for this resource, and convert to a face in
     XEmacs rather than a font in Xt/Xft.
          ! DEPRECATED resource xftFont.
          ! To be retargeted to an Xt converter which returns a font.
          !XEmacs*Tabs.xftFont: Bitstream Vera Sans-16
          !XEmacs*menubar*xftFont: Bitstream Vera Sans-16
          XEmacs*Tabs.fcFontName: Bitstream Vera Sans-16
          XEmacs*menubar*fcFontName: Bitstream Vera Sans-16
          XEmacs.modeline.attributeFont: Bitstream Charter-16
          XEmacs.default.attributeFont: Bitstream Vera Sans Mono-16
     I highly recommend use of a proportional font in the modeline
     because it allows a lot more text to fit there.  (Previously the
     font sizes were quite varied, and there was a comment that this
     weirdness gave good balance.  This isn't true on my main platform,
     Mac OS X, and needs to be rechecked on Linux, where it was
     observed.)  Note that you can probably specify a particular
     Japanese font with something like
          XEmacs.default.attributeFont: Bitstream Vera Sans Mono,Sazanami Mincho-16
     Order is important; Japanese fonts will support English, but
     Sazanami's Roman characters are not very pretty compared to the
     Bitstream font.  *NOTE*: _This is untested, but should work in
     theory._


File: internals.info,  Node: Better Rendering Support -- Modern Font Support,  Prev: Better Rendering Support -- Configuration with the Interim Patches,  Up: Future Work -- Better Rendering Support

44.22.5 Better Rendering Support - Modern Font Support
------------------------------------------------------

*NB: This subtree eventually needs to be moved to the Lispref.*

   This chapter describes integration of the `Xft' font support library
into XEmacs.  This library is a layer over the separate `FreeType'
rendering engine and `fontconfig' font query and selection libraries.
`FreeType' provides rendering facilities for modern, good-looking
TrueType fonts with hinting and antialiasing, while `fontconfig'
provides a coherent interface to font query and selection which is
independent of the rendering engine, although currently it is only used
in `Xft' to interface to `FreeType'.

   From the user's point of view, `fontconfig' provides a naming
convention which is _precise_, _accurate_, and _convenient_.  Precision
means that all properties available in the programming API can be
individually specified.  Accuracy means that the truename of the font
is exactly the list of all properties specified by the font.  Thus, the
anomalies that occur with XLFDs on many servers (including modern Linux
distributions with XFree86 or X.org servers) cannot occur.  Convenience
is subjective, of course.  However, `fontconfig' provides a
configuration system which (1) explicitly specifies the defaults and
substitutions that will be made in processing user queries, and (2)
allows the user to specify search configuration, abbreviations,
substitutions, and defaults that override the system's, in the same
format as used by system files.  Further, a "standard minimal
configuration" is defined that ensures that at least _serif_,
_sans-serif_, and _monospace_ fonts are available on all `fontconfig'
systems.

* Menu:

* Modern Font Support -- Font Concepts::  GUI devices, fonts, glyphs, rendering.
* Modern Font Support -- fontconfig::  Querying and selecting fonts.
* Modern Font Support -- Xft::  Rendering fonts on X11.


File: internals.info,  Node: Modern Font Support -- Font Concepts,  Next: Modern Font Support -- fontconfig,  Prev: Better Rendering Support -- Modern Font Support,  Up: Better Rendering Support -- Modern Font Support

44.22.5.1 Modern Font Support - Font Concepts
.............................................

In modern systems, displays are invariably "raster graphic devices",
which present an abstract interface of "pixel array" where each "pixel
value" is a color, and each pixel is individually mutable, and
(usually) readable.  In XEmacs, such devices are collectively called
"GUI devices", as opposed to "TTY devices" which are character stream
devices but may support control sequences for setting the color of
individual characters, and the insertion position in a rectangular
array.  Here we are concerned only with control of GUI devices but use
TTY devices as a standard for comparison.

   A "font" is an indexed collection of "glyphs", which are
specifications of character shapes.  On a TTY device, these shapes are
entirely abstract, and the index is the identity function.  Typically
fonts are embedded in TTY devices, the user has no control over the font
from within the application, and where choice is available, there is
limited selection, and no extensibility.  Simple, functional, and ...
ugly.

   On GUI devices, the situation is different in every respect.  Glyphs
may be provided by the device, the application, or the user.  Additional
glyphs may be added at will at any of those levels.  Arbitrary index
functions allow the same glyph to be used to display characters in
different languages or using application-specific codes.  Glyphs have
concrete APIs, allowing fine control of rendering parameters, even
user-specified shapes.  To provide convenient, consistent handling of
collections of glyphs, we need a well-defined font API.

   We can separate the necessary properties into two types: properties
which are common to all glyphs in the collection or a property of the
collection itself, and those which are glyph-specific.  Henceforth, the
former are called "font properties" and the latter "glyph properties".

   Font properties include identification like the font "family",
font-wide design parameters like "slant" and "weight", font "metrics"
like "size" (nominal height) and "average width" used for approximate
layout (such as sizing a popup dialog), and properties like the
"default glyph" that are associated with the font for convenient use by
APIs, but aren't really an intrinsic property of the font as a
collection of glyphs.  There may also be a "kerning table" (used to
improve spacing of adjacent glyphs).

   Glyph properties include the "index", glyph metrics such as
"ascent", "descent", "width", "offset" (the offset to the normal
position of the next glyph), "italic correction" (used to improve
spacing when slanted and unslanted glyphs are juxtaposed).  Most
important, of course, is the glyph's shape, which is provided in a
format specific to a rendering engine.  Common formats include bitmaps
(X11 BDF), Postcript programs (Type 1), and collections of spline curves
(TrueType).  When the shape is not itself a bitmap, it must be
"rendered" to a pixmap, either a region on the display or a separate
object which is copied to the display.  In that case, the shape may
include "multiple masters" or "hints" to allow context-specific
rendering which improves the appearance of the glyph on the display.

   Note that this use of "glyph" is mostly independent of the XEmacs
LISP glyph API.  *Note Glyphs::.  It is possible to extract a single
glyph from a font and encapsulate it in Lisp_Glyph object, but the LISP
glyph API allows access to only a very few glyph properties, none of
them related to the rendering process.

   XEmacs LISP does provide an API for selecting and querying fonts, in
the form of a fairly complete set of wrappers for `fontconfig' (*note
Modern Font Support -- fontconfig::).  It also provides some control of
rendering of text via wrappers for `Xft' APIs (*note Modern Font
Support -- Xft::), but this API is quite incomplete.  Also, since the
font selection and query facilities of `Xft' are provided by
`fontconfig', there is some confusion in the API.  For example, use of
antialiasing to improve the appearance of rendered glyphs can be
enabled or disabled.  The API for this is to set the `fontconfig' font
property `antialias' on the font.  However, from the point of view of
`fontconfig' this is merely a hint that the rendering engine may or may
not respect.  This property cannot be used to select only fonts
suitable for being antialiased, for example.  And `rgba' (subpixel
geometry) and `dpi' (pixel density) are conceptually properties of the
display, not of either the font.  They function as hints to the
rendering process.

   As a final confusing touch, `Xft' also provides some access to the
`XRender' extension provided by some modern X servers.  This is mostly
limited to colors, but rectangle APIs are also provided.  These are (of
course) completely independent of fonts, but `Xft' is designed for
client-side font rendering, and thus uses the `XRender' extension
heavily.


File: internals.info,  Node: Modern Font Support -- fontconfig,  Next: Modern Font Support -- Xft,  Prev: Modern Font Support -- Font Concepts,  Up: Better Rendering Support -- Modern Font Support

44.22.5.2 Modern Font Support - fontconfig
..........................................

_Implementation notes_: The functions which initialize the library and
handle memory management (_e.g._, `FcInit' and `FcPatternDestroy') are
intentionally not wrapped (in the latter case, `fc-pattern-destroy' was
provided, but this was ill-considered and will be removed; LISP code
should *never* call this function).  Thinking about some of the
auxiliary constructs used by `fontconfig' is in transition.  The
`FcObjectSet' API has been internalized; it is exposed to LISP as a
list of strings.  The `FcFontSet' API is still in use, but it also will
be internalized, probably as a list (alternatively, vector) of
`Lisp_fc_pattern' objects.  Changing the representation of `fontconfig'
objects (property names) from LISP strings to keywords is under
consideration.

   If `Xft' (including `fontconfig') support is integrated into the
XEmacs build, XEmacs provides the symbol `xft' at initialization.

   XEmacs provides the following functions wrapping the `fontconfig'
library API.

 -- Function: fc-fontset-p object
     Returns t if OBJECT is of type fc-fontset, nil otherwise.  _This
     API is likely to be removed in the near future._

 -- Function: fc-fontset-count fcfontset
     Counts the number of fc pattern objects stored in the fc fontset
     object FCFONTSET.  _This API is likely to be removed in the near
     future._

 -- Function: fc-fontset-ref fcfontset i
     Return the fc pattern object at index I in fc fontset object
     FCFONTSET.  Return nil if the index exceeds the bounds of
     FCFONTSET.  _This API is likely to be removed in the near future._

 -- Function: fc-fontset-destroy fcfontset
     Explicitly deallocate FCFONTSET.  _Do not call this function from
     LISP code.  You will crash.  This API will be removed in the near
     future._

 -- Function: fc-pattern-p object
     Returns t if OBJECT is of type fc-pattern, nil otherwise.

 -- Function: fc-pattern-create
     Return a fresh and empty fc-pattern object.

 -- Function: fc-name-parse name
     Parse string NAME as a fontconfig font name and return its
     representation as a fc pattern object.

 -- Function: fc-name-unparse pattern
     Unparse pattern object PATTERN to a string.

   `Xft''s similar function is actually a different API.  We provide
both for now.  (They probably invoke the same code from `fontconfig'
internally, but the `fontconfig' implementation is more conveniently
called from C.)

 -- Function: xft-name-unparse pattern
     Unparse pattern object PATTERN to a string (using the `Xft' API).

 -- Function: fc-pattern-duplicate pattern
     Make a copy of pattern object PATTERN and return it.

 -- Function: fc-pattern-add pattern property value
     Add attributes to the pattern object PATTERN.  PROPERTY is a
     string naming the attribute to add, VALUE the value for this
     attribute.

     VALUE may be a string, integer, float, or symbol, in which case
     the value will be added as an FcChar8[], int, double, or FcBool
     respectively.

 -- Function: fc-pattern-del pattern, property
     Remove attribute PROPERTY from pattern object PATTERN.

   This is the generic interface to `FcPatternGet'.  We don't support
the losing symbol-for-property interface.  However, it might be a very
good idea to use keywords for property names in LISP.

 -- Function: fc-pattern-get pattern property &optional id type
     From PATTERN, extract PROPERTY for the ID'th member, of type TYPE.

     PATTERN is an `Xft' (`fontconfig') pattern object.  PROPERTY is a
     string naming a `fontconfig' font property.  Optional ID is a
     nonnegative integer indexing the list of values for PROPERTY
     stored in PATTERN, defaulting to 0 (the first value).  Optional
     TYPE is a symbol, one of 'string, 'boolean, 'integer, 'float,
     'double, 'matrix, 'charset, or 'void, corresponding to the FcValue
     types.  ('float is an alias for 'double).

     Symbols with names of the form `fc-result-DESCRIPTION' are
     returned when the desired value is not available.  These are

          fc-result-type-mismatch   the value found has an unexpected type
          fc-result-no-match        there is no such attribute
          fc-result-no-id           there is no value for the requested ID

   The Lisp types returned will conform to TYPE:

     string          string
     boolean         `t' or `nil'
     integer         integer
     double (float)  float
     matrix          not implemented
     charset         not implemented
     void            not implemented

   The types of the following standard properties are predefined by
fontconfig.  The symbol 'fc-result-type-mismatch will be returned if the
object exists but TYPE does not match the predefined type.  It is best
not to specify a type for predefined properties, as a mistake here
ensures error returns on the correct type.

   Each standard property has a convenience accessor defined in
`fontconfig.el', named in the form `fc-pattern-get-PROPERTY'.  The
convenience functions are preferred to `fc-pattern-get' since a typo in
the string naming a property will result in a silent null return, while
a typo in a function name will usually result in a compiler or runtime
\"not fboundp\" error.  You may use `defsubst' to define convenience
functions for non-standard properties.

     family         String  Font family name
     style          String  Font style. Overrides weight and slant
     slant          Int     Italic, oblique or roman
     weight         Int     Light, medium, demibold, bold or black
     size           Double  Point size
     aspect         Double  Stretches glyphs horizontally before hinting
     pixelsize      Double  Pixel size
     spacing        Int     Proportional, monospace or charcell
     foundry        String  Font foundry name
     antialias      Bool    Whether glyphs can be antialiased
     hinting        Bool    Whether the rasterizer should use hinting
     verticallayout Bool    Use vertical layout
     autohint       Bool    Use autohinter instead of normal hinter
     globaladvance  Bool    Use font global advance data
     file           String  The filename holding the font
     index          Int     The index of the font within the file
     ftface         FT_Face Use the specified FreeType face object
     rasterizer     String  Which rasterizer is in use
     outline        Bool    Whether the glyphs are outlines
     scalable       Bool    Whether glyphs can be scaled
     scale          Double  Scale factor for point->pixel conversions
     dpi            Double  Target dots per inch
     rgba           Int     unknown, rgb, bgr, vrgb, vbgr, none - subpixel geometry
     minspace       Bool    Eliminate leading from line spacing
     charset        CharSet Unicode chars encoded by the font
     lang           String  List of RFC-3066-style languages this font supports

   The FT_Face, Matrix, CharSet types are unimplemented, so the
corresponding properties are not accessible from Lisp at this time.  If
the value of a property returned has type FT_Face, FcCharSet, or
FcMatrix, `fc-result-type-mismatch' is returned.

   The following properties which were standard in `Xft' v.1 are
obsolete in `Xft' v.2: `encoding', `charwidth', `charheight', `core',
and `render'.

 -- Function: fc-pattern-destroy pattern
     Explicitly deallocate pattern object PATTERN.  _Do not call this
     function from LISP code.  You will crash.  This API will be removed
     in the near future._

 -- Function: fc-font-match device pattern
     Return the font on DEVICE that most closely matches PATTERN.

     PATTERN is a `fontconfig' pattern object.  DEVICE is an X11
     device.  Returns a `fontconfig' pattern object representing the
     closest match to the given pattern, or an error code.  Possible
     error codes are `fc-result-no-match' and `fc-result-no-id'.

 -- Function: fc-list-fonts-pattern-objects device pattern properties
     List the fonts on DEVICE that match PATTERN for PROPERTIES.
     DEVICE is an X11 device.  PATTERN is a `fontconfig' pattern to be
     matched.  PROPERTIES is the list of property names (strings) that
     should be included in each returned pattern.  The result is a
     `fontconfig' fontset object containing the set of unique matching
     patterns.

   The PROPERTIES argument does not affect the matching.  So, for
example,

     (mapcar #'fc-name-unparse
       (let ((xfl (fc-list-fonts-pattern-objects nil
                   (fc-name-parse "FreeMono") '("style")))
             (i 0)
             (fl nil))
         (while (< i (fc-fontset-count xfl))
           (push (fc-fontset-ref xfl i) fl)
           (setq i (1+ i)))
         fl))

   will return something like `(":style=Bold" ":style=Medium"
":style=Oblique" ":style=BoldOblique")' if you have the FreeFont
package installed.  Note that the sets of objects in the target pattern
and the returned patterns don't even intersect.

   In using `fc-list-fonts-pattern-objects', be careful that only
intrinsic properties of fonts be included in the pattern.  Those
properties included in the pattern must be matched, or the candidate
font will be eliminated from the list.  When a font leaves a property
unspecified, it is considered to be a mismatch for any pattern with that
property specified.  Thus, inclusion of extraneous properties will
result in the list being empty.  Note that for scalable fonts (at
least), `size' is not an intrinsic property!  Thus a specification such
as `"Bitstream Vera Sans-12"' will return an empty list regardless of
whether the font is available or not--probably not what you (as
programmer or user) want.

   The list is unsorted.  In particular, the pattern
`":style=italic,oblique"' will not return italic fonts first, then
oblique ones.  The fonts will be returned in some arbitrary order.

   _Implementation notes_:  Fontset objects are slated for removal from
the API.  In the future `fc-list-fonts-pattern-objects' will return a
list.  The DEVICE argument is unused, ignored, and may be removed if
it's not needed to match other font-listing APIs.  This name will be
changed to correspond to Ben's new nomenclature, probably simply
`fc-font-list'.

 -- Function: fc-font-sort device pattern trim
     Return a fontset object listing all fonts sorted by proximity to
     PATTERN.  DEVICE is an X11 device.  PATTERN is a fontconfig
     pattern to be matched.  Optional argument TRIM, if non-nil, means
     to trim trailing fonts that do not contribute new characters to
     the union repertoire.

   _Implementation notes_:  Fontset objects are slated for removal from
the API.  In the future `fc-font-sort' will return a list (or perhaps a
vector) of FcPatterns.  The DEVICE argument is unused, ignored, and may
be removed if it's not needed to match other font-listing APIs.

 -- Function: fc-font-real-pattern fontname xdevice
     Temporarily open font FONTNAME (a string) on device XDEVICE and
     return the actual fc pattern matched by the Fc library.  _This
     function doesn't make much sense and will be removed from the API._

 -- Function: xlfd-font-name-p fontname
     Check whether string FONTNAME is a XLFD font name.

 -- Variable: xft-debug-level
     Level of debugging messages to issue to stderr for XFT.  A
     nonnegative integer.  Set to 0 to suppress all warnings.  Default
     is 1 to ensure a minimum of debugging output at initialization.
     Higher levels give more information.

 -- Variable: xft-version
     The major version number of the Xft library compiled with.

 -- Variable: xft-xlfd-font-regexp
     Regular expression matching XLFD font names.


File: internals.info,  Node: Modern Font Support -- Xft,  Prev: Modern Font Support -- fontconfig,  Up: Better Rendering Support -- Modern Font Support

44.22.5.3 Modern Font Support - fontconfig
..........................................

IIRC, we don't really provide any `Xft' APIs at the LISP level yet.


File: internals.info,  Node: Future Work Discussion,  Next: Old Future Work,  Prev: Future Work,  Up: Top

45 Future Work Discussion
*************************

This chapter includes (mostly) email discussions about particular design
issues, edited to include only relevant and useful stuff.  Ideally over
time these could be condensed down to a single design document to go
into the normal Future Work section.

* Menu:

* Discussion -- Garbage Collection::
* Discussion -- Glyphs::
* Discussion -- Dialog Boxes::
* Discussion -- Multilingual Issues::
* Discussion -- Instantiators and Generic Property Accessors::
* Discussion -- Switching to C++::
* Discussion -- Windows External Widget::
* Discussion -- Packages::
* Discussion -- Distribution Layout::


File: internals.info,  Node: Discussion -- Garbage Collection,  Next: Discussion -- Glyphs,  Prev: Future Work Discussion,  Up: Future Work Discussion

45.1 Discussion - Garbage Collection
====================================

* Menu:

* Discussion -- KKCC::
* Discussion -- Incremental Collector::
* Discussion -- Pure Space::
* Discussion -- Hashtable-Based Marking and Cleanup::
* Discussion -- The Anti-Cons::


File: internals.info,  Node: Discussion -- KKCC,  Next: Discussion -- Incremental Collector,  Prev: Discussion -- Garbage Collection,  Up: Discussion -- Garbage Collection

45.1.1 Discussion - KKCC
------------------------

KKCC is the tag used for the "new garbage collector algorithms," which
are a refactoring of the garbage collector to make trying new collectors
simpler.


File: internals.info,  Node: Discussion -- Incremental Collector,  Next: Discussion -- Pure Space,  Prev: Discussion -- KKCC,  Up: Discussion -- Garbage Collection

45.1.2 Discussion - Incremental Collector
-----------------------------------------

The incremental collector was an attempt by Marcus Crestani to achieve
better GC performance by saving the state of the GC and returning
control to the Lisp interpreter partway through collection, resuming
collection after a certain number of further allocations. In order to
implement this it was necessary to use memory protection (e.g.
mprotect(2)) to determine if objects that had already been processed
for GC were modified by Lisp (or indeed other code) between cycles of
incremental collection.

   Unfortunately this meant that if memory not under the control of the
runtime allocator (e.g. Lisp_Subr structures (which are static C data),
the dumped Lisp generated by pdump) could possibly contain references
to Lisp objects, then that memory needed to be copied to objects under
the control of the runtime allocator (mc-alloc in that implementation),
in order for the memory protection to work. This increased memory usage
and startup time.

   Further complicating the picture was that the redisplay code was not
careful about GC protection of its glyph data. This did not matter
before incremental GC was introduced, since redisplay could just say
"don't GC in redisplay, please." KKCC actually addresses this problem,
in that with old-GC it marks those Lisp objects reachable via the
redisplay structures. But, again, redisplay does not allocate its
objects via Marcus' code, and so intermediate structures are not
available to the memory protection and the "tricolor marking" he
implements. This would have been fixable.

   However, Marcus documents (see
https://crestani.de/xemacs/pdf/mc-alloc.pdf ) that the incremental
allocator is slower and uses more memory than the old allocator. This
accords with my personal experience, of increased hangs and subjective
slowdowns, possibly because of the overhead of mprotect(2) during
incremental collection. E.g. (profile-expression (hanoi 9)) gives a 20%
slow down in an optimized build with incremental collection turned off,
most of this in GC, and redisplay stutters more in my assessment.

   For this reason I (Aidan Kehoe) removed the incremental collection
code in October 2021, after discussion with Stephen Turnbull and Mats
Lidell.


File: internals.info,  Node: Discussion -- Pure Space,  Next: Discussion -- Hashtable-Based Marking and Cleanup,  Prev: Discussion -- Incremental Collector,  Up: Discussion -- Garbage Collection

45.1.3 Discussion - Pure Space
------------------------------

On Tue, Oct 12, 1999 at 03:36:59AM -0700, Ben Wing wrote:

   So what am I missing here?

   In response, Olivier Galibert wrote:

   Two things:
  1. The purespace is gone

     I  mean  absolutely, completely and utterly  removed.   Fpurecopy
     is a no-op now (and  have been for  some time).  Readonly objects
     are gone too.  Having  less checks to  do in Fsetcar,  Fsetcdr,
     Faset and some others  is probably a  good thing, speedwise.  I
     have it removed some time ago because it  does not make  sense
     when using a portable dumper to copy data in a special area of the
     memory at dump time and I wanted to be  sure that supressing the
     copying  from Fpurecopy wouldn't break things.

     Now, we want to get the post-dumping data sharing back, of course.
     In today systems,  it is  quite   easy: you just   have  to map
     the  file MAP_PRIVATE and avoid writing to the subset of  pages
     you want to keep shared.   Copy-on-write does  the job for  you.
     It  has the nice side effect of  completely avoiding bus  errors
     due  to trying to  write to readonly memory zones.

     Avoiding writing to the "pure" objects themselves  is already
     done, of course.  Would lisp code  have written to the  purecopied
     parts of the dumped data that it would have exploded long ago.  So
     there is nothing to do in this area.  So the only remaining thing
     is  the markbit.  Two possible strategies:

        * have Fpurecopy mark somehow the lrecords it would have copied
          in the good old times.  Post-dump, use this mark as a "always
          marked, don't touch, don't look  into, don't free"  flag, the
          same way CHECK_PURE was used.

        * move the markbit outside of the lrecord.

     The second solution is more appealing to me for a bunch of reasons:
        * more things are shared  than only what  is purecopied (not
          yet  used functions come to mind)

        * no more "the only references to this  non-purecopied object
          are from purecopied objects, XEmacs will self-destruct  in
          ten seconds"  kind of bugs.

        * removing flags  goes   the right   way towards   implementing
          Jan's allocator ideas.

        * it becomes probably easier to experiment with the GC code

  2. Finding all the dumped objects in order to unmark them sucks

     Not  having to rebuild  a list of all the  dumped objects  in
     order to find  them all and ensure that  all are unmarked
     simplifies things for me.  Errr, ok, now that I really think of
     it, I can rebuild this list easily, in fact.  And I'm probably
     going to have to manage it, since I feel like the  lack of calls
     to  the finalizers for the dumped objects is going to someday
     turn over and bite me  in the face.  But anyways, it makes my life
     easier for now.

     So no,  it's  not a _necessity_.   But  it  helps.  And the
     automatic sharing of  all objects until  you write  to   them
     explicitely is,  I think, really cool.


File: internals.info,  Node: Discussion -- Hashtable-Based Marking and Cleanup,  Next: Discussion -- The Anti-Cons,  Prev: Discussion -- Pure Space,  Up: Discussion -- Garbage Collection

45.1.4 Discussion - Hashtable-Based Marking and Cleanup
-------------------------------------------------------

On 10/12/1999 5:49 PM Ben Wing wrote:

   OK, I can see the advantages.  But:

  1. There will be an inevitable loss of speed using a large hashtable.
     If it's large, I say that it's just not worth it.  There are
     things that are so much more important than futzing around with
     the garbage collector (e.g. fixing the god damn user interface),
     things which if not fixed will sooner or later cause XEmacs to die
     entirely.  If we are causing a major slowdown in the name of some
     not-so-important work that may or may not get done, we shouldn't
     do it.  (On the other hand, if the slowdown is negligible, I have
     no problems with this.)

  2. I think you should *expand* the concept of read-only objects so
     that *any* object (especially strings and cons cells) can get
     marked read-only by the C code if it wants. (Perhaps you could use
     the now-unused mark bit to hold a read-only flag.) This is
     important because it allows C code to directly return internal
     lists (e.g. from the specifiers and various object property lists)
     without having to do a copy, like is now done (and similarly,
     potentially to directly accept lists from a Lisp call without
     copying them for internal use, if the Lisp caller is made aware
     that the list might become read-only) - if the copy weren't done
     and some piece of Lisp code went and modified the list, XEmacs
     might very well crash.  Thus, this read-only flag would be a huge
     efficiency gain in terms of the garbage collection overhead saved
     as well as the speed of copying a large list.  The extra checks in
     `Fsetcar()', etc. for this that you mention are in fact negligible
     in their speed overhead - one or two instructions - and these
     functions are not used all that commonly, either.  With the
     changes I have proposed in Architecting XEmacs, the case of
     returning an internal list will become more and more common as the
     power of the user interface would be greatly increased and along
     with it are lots and lots of lists of info that need to be
     retrievable from Lisp.

   BTW there is a wonderful book all about garbage collection by Jones
and Lins.  Ever seen it?

     http://www.amazon.com/exec/obidos/ASIN/0471941484/qid=939775572/sr=1-1/002-3092633-2509405


File: internals.info,  Node: Discussion -- The Anti-Cons,  Prev: Discussion -- Hashtable-Based Marking and Cleanup,  Up: Discussion -- Garbage Collection

45.1.5 Discussion - The Anti-Cons
---------------------------------

From: "Ben Wing" <ben@666.com> Date: Tue, 14 May 2002 06:48:09 -0700

   i was thinking about the proliferating types of weak hash tables -
e.g. now we have "key-car-value weak" hash tables due to a need in the
glyphs code.  i realized there should be a general solution, that lets
you control exactly how the weakness of such hash tables work.

   and, assuming we implement a simple "reference" type, a simple
container whose object is a weak reference and thus gets converted to
nil (and a flag set on the reference) when the object is collected, it
would be useful for more precisely controlling the reference, too.

   it's called an "anti-cons".  it behaves somewhat like a cons in that
it boxes two items, but its marking properties are very different - in
fact, backwards.  normally, a cons, if marked, marks its children.  in
this case, if the children of an anti-cons are marked, it marks itself!
you'd need a few different kinds of anti-cons - probably the following:

     and [marks itself if both children marked]
     or [...]
     left [marks itself if left is marked, and then marks the right]
     right [...]
     not-left
     not-right

   by putting such an object inside of a weak reference - e.g. in a weak
hash table - we can set up a tree of arbitrary complexity which
implements any boolean formula of markedness over any number of
objects.  this would easily handle key-car, and key-cadr, and
key-car-or-cdr, and key-((caar or cadr) and cdr) etc. etc.

   implementing this in the current xemacs framework is mostly trivial.

   michael, would such an object get in the way of your new gc?

   From: sperber@informatik.uni-tuebingen.de (Michael Sperber [Mr.
Preprocessor]) Date: Tue, 14 May 2002 16:04:01 +0200

   You might want to look at

   http://research.microsoft.com/Users/simonpj/Papers/weak.htm

   for a pretty comprehensive survey of what you could want in terms of
weakness.  Its weak pointers are very similar to your anti-cons.
However, there are some problems in doing the same in a Lisp settings,
mainly because of symbols.  I intend to elaborate on this next week;
this week is full, unfortunately.

   Ben> implementing this in the current xemacs framework is mostly
Ben> trivial.

   Ben> michael, would such an object get in the way of your new gc?

   Well, our first commit will be an implementation of vanilla weak
boxes (ready within the next few days, I hope), and we'll then try to
replace most other instances of weakness with uses of those.  We'll
then try to find a more general solution for the rest.  (Richard
Reingruber has already done a comprehensive survey of the trouble spot.

   Can you wait until next week?  I'll try to come up with a battle plan
then.

   From: sperber@informatik.uni-tuebingen.de (Michael Sperber [Mr.
Preprocessor]) Date: Tue, 28 May 2002 16:14:20 +0200

   We've now started implementing ephemerons as a building block for the
more involved weakness-involving data structures:

   The relevant reference is

   Barry Hayes. Ephemerons: A New Finalization Mechanism.  OOPSLA 1997.
176-183

   The idea is this:

   an ephemeron consists of a key and a value.  Through the ephemeron,
the key is not reachable.  The value is only reachable if both the
ephemeron is reachable and the key is reachable.  If the ephemeron is
reachable and the key becomes unreachable, the value slot of the
ephemeron will be tombstoned, i.e. overwritten with NIL or something.

   This allows implementing, AFAICS, the other data structures involving
weakness, such as weak hash tables and their various mutants.

   We're also planning to come up with a more comprehensive solution for
finalization, but some design snags remain to be worked out.


File: internals.info,  Node: Discussion -- Glyphs,  Next: Discussion -- Dialog Boxes,  Prev: Discussion -- Garbage Collection,  Up: Future Work Discussion

45.2 Discussion - Glyphs
========================

Some comments (not always pretty!) by Ben:

   March 20, 2000

   Andy, I use the tab widgets but I've been having lots of problems.

   1] Sometimes clicking on them does nothing.

   2] There's a design flaw: I frequently use M-C-l to switch to the
previous buffer.  If I use this in conjunction with the tabs, things get
all screwed up because selecting a buffer with the tab does not bring it
to the front of the buffer list, like it should.  It looks like you're
doing this to avoid having the order of the tabs change, but this is
wrong: If you don't reorder the buffer list, everything else gets
screwed up.  If you want the order of the tabs not to change, you need
to decouple this order from the buffer list order.

   March 23, 2000

   I'm very confused.  The SIGIO timer is used *only* for C-g.  It has
nothing to do with any other events.  (sit-for 0) ought to

   (1) cause all pending non-command events to get executed, and (b) do
redisplay

   However, sit-for gets preempted by input coming in.

   What about (sit-for 0.1)?

   I suppose a solution along the lines of dispatch-non-command-events
might be OK if you've tried everything else and it doesn't work, but i'm
leery of introducing new Lisp functions to deal with specific problems.
Pretty soon we end up with a whole bevy of such ill-defined functions,
like we already have.  I think instead, you should introduce the
following primitive:

     (wait-for-event redisplay &rest event-specs)

   Waits for one of the event specifications specified to happen.
Returns something about what happened.

   REDISPLAY controls the behavior of redisplay during waiting.
Something like

   * nil (never redisplay),

   * t (redisplay when it seems appropriate), etc.

   EVENT-SPECS could be

     t                     -- drain all non-user events, and then return
     any-process           -- wait till input or state change on any process
     process               -- wait till input or state change on process
     time                  -- wait till such-and-such time has elapsed
     'user                 -- wait till user event has happened
     '(user predicate)     -- wait till user event matching the predicate has
                              happened
     'event                -- wait till any event has happened
     '(event predicate)    -- wait till event matching the predicate has happened

   The existing functions `next-event', `next-command-event',
`accept-process-output', `sit-for', `sleep-for', etc. could all be
written in terms of this new command.  You could use this command inside
of your glyph code to ensure that the events get processed that need do
in order for widget updates to happen.

   But you said something about need a magic event to invoke redisplay?
Why is that?

   April 2, 2000

   the internal distinction between "widget" and "layout" is bogus.
there exist widgets that do drawing and do layout of their children,
e.g. group-box widgets and proper tab widgets.  the only sensible
distinction is between widgets with children and those without children.

   April 5, 2000

   andy, i'm not sure i really believe that you need to cycle the event
code to get widgets to redisplay, but in any case you should

  1. hide the logic to do this in the c code; the lisp code should do
     nothing other than call (redisplay widget)

  2. make sure your event-cycling code processes *NO* events at all.
     this includes non-user events.  queue the events instead.

   in other words, dispatch-non-command-events must go, and i am
proposing a general function (redisplay OBJECT) to replace the existing
ad-hoc functions.

   April 6, 2000

   the tab widget code should simply be able to create a whole lot of
tabs without regard to the size of the gutter, and the surrounding
layout widget (please please make layouts be proper widgets!) should
automatically map and unmap them as necessary, to fill up the available
space.  perhaps this already works and what you're doing is just for
optimization?  but i get the feeling this is not the case.

   April 6, 2000

   the function make-gutter-only-dialog-frame is bogus.  the use of the
gutter here to hold widgets is an implementation detail and should not
be exposed in the interface.  similarly, make-search-dialog should not
have to do all the futzing that it does.  creating the frame unmapped,
creating an extent and messing with the gutter: all this stuff should be
hidden.  you should have a simple function make-dialog-frame that takes
a dialog specification, and that's all you need to do.

   also, these dialog boxes, and this function make-dialog-frame, should

  1. be in `dialog.el', not gutter-items.el.

  2. when possible, be placed in the interactive spec of standard lisp
     functions rather than accessed directly from `menubar-items.el'

  3. wrapped in calls to should-use-dialog-box-p, so the user has
     control over when dialog boxes appear.

   April 7, 2000

   hmmm ...  in that case, the whitespace absolutely needs to be
specified as properties of the layout widget (e.g. :border-width and
:border-height), rather than setting an overall size.  you have no idea
what the correct size should be if the user changes font size or uses
translations in a different language.

   Your modus operandi should be "hardcoded pixel sizes are *always*
bad."

   April 7, 2000

   you mean the number of tabs adjusts, or the size of each tab adjusts
(by making the font smaller or something)?  if the size of a single tab
is not related to the total space the tabs can fix into, then it should
be possible to simply specify as many tabs as exist for buffers, and
have the layout manager decide how many can fit into the available
space.  this does *not* mean the layout manager will resize the tabs,
because query-geometry on the tabs should find out that the tabs don't
want to be any size other than they are.

   the point here is that you should not *have* to worry about pixel
heights and widths *anywhere* in Lisp-level code.  The layout managers
should take care of everything for you.  The only exceptions may be in
some text fields, which will be blank by default and you want to specify
a maximum width (which should be done in 'n' sizes, not in pixels!).

   i won't stop complaining until i see nearly every one of those
pixel-width and pixel-height parameters gone, and the remaining ones
there for a very, very good reason.

   April 7, 2000

   Andy Piper wrote:

     > At 03:51 PM 4/6/00 -0700, Ben Wing wrote:
     > >[the function make-gutter-only-dialog-frame is bogus]
     >
     > The problem is that some of the callbacks and such need access to the
     > *created* frame, so you end up in a catch 22 unless you do what I've done.

   [Ben proposes other ways to avoid exposing all the guts, as in
`make-gutter-only-dialog-frame':]

  1. Instead of passing in the actual glyph spec or glyph, pass in a
     function of two args (the dialog frame and its parents), which when
     called, creates and returns the appropriate glyph.

  2. [Better] Provide a way for callbacks to determine where they were
     invoked at.  This is much more general and is what you should
     really do.  For example, have the code that calls the callbacks
     bind some global variables such as widget-callback-current-glyph
     and widget-callback-current-channel, which contain the glyph whose
     callback is being invoked, and the window or frame of the glyph
     (depending on where the glyph is) where the invocation actually
     happened.  That way, the callbacks can easily figure out the dialog
     box and its parent, and not have to worry about embedding it in at
     creation time.

   April 15, 2000 I don't understand when you say "the various types of
callback".  Are you using the callback for various different purposes?

   Your widget callbacks should work just like any other callback: they
take two arguments, one indicating the object to which the callback was
attached (an image instance, i think), and the event that caused the
callback to be invoked.

   April 17, 2000

   I am completely vetoing widget-callback-current-channel.  How about
you create a new keyword, :new-callback, that is a function of two args,
like i specified before.

   btw if you really are calling your callback using call-interactively,
why don't you declare a function (interactive "e") and then call
event-channel on the resulting event?  that should get you the same
result as widget-callback-current-channel.

   the problem with this and everything you've proposed is that there's
no way, of course, to get at the actual widget that you were invoked
from.  would you propose adding widget-callback-current-widget?


File: internals.info,  Node: Discussion -- Dialog Boxes,  Next: Discussion -- Multilingual Issues,  Prev: Discussion -- Glyphs,  Up: Future Work Discussion

45.3 Discussion - Dialog Boxes
==============================

     From:
             Ben Wing <ben@666.com>
                                                           10/7/1999 5:57 PM

      Subject:
             Re: Animated gif patch (2)
          To:
             Andy Piper <andy@xemacs.org>
         CC:
             xemacs-review@xemacs.org, xemacs-beta@xemacs.org




     The distinction between layouts and widgets makes no sense, so you should combine
     the different data required.  Consider a grouping widget.  Is this a layout or a
     widget?  It draws, like a widget, but has children, like a layout.  Same for a tab
     widget, properly implemented.  It draws, handles input, has children, and makes
     choices about how to lay them out.

     ben

     From:
             Ben Wing <ben@666.com>
                                                            9/7/1999 8:50 PM

      Subject:
             Re: Layouts done
          To:
             Andy Piper <andyp@beasys.com>




     this sounds great!  where can i see the code?

     as for user-defined layouts, you must certainly have some sort of abstraction
     layer for layouts, with DEFINE_LAYOUT_TYPE or something similar just like device
     types and such.  If not, you should certainly make one ...  it would have methods
     such as query-geometry and do-layout.  It should be easy to create a user-defined
     layout if you have such an abstraction.

     with a user-defined layout, complex built-in layouts such as grid should not be
     necessary because it's so easy to write snippets of lisp.

     as for the "redisplay too much" problem, perhaps you could put a dirty flag in
     each glyph indicating whether it needs to be redisplayed, recalculated, etc.?

     Andy Piper wrote:

     > You may want to check them out. I haven't done the user-defined layout
     > callback - I'm not sure what sort of API this could have.  Keywords I've done:
     >
     > :orientation - vertical or horizontal
     > :justify - left, center or right
     > :border - etch-in, etch-out, bevel-in, bevel -out or text (which gives you
     > etch-in with a title)
     >
     > You can embed any glyph type in a layout.
     >
     > There is probably room for improvements for justify to do grid-type layouts
     > as per java.
     >
     > The only annoying thing is that I've hacked up font-lock support to do a
     > progress gauge in the gutter area. I've used a layout to set things out
     > correctly. The problem is if you change one of the sub-widgets, the whole
     > layout gets redisplayed because it is treated as a single glyph by redisplay.
     >
     > Oh, and I've done line based scrolling so that glyphs scroll off the page
     > in units of the average display line height rather than the whole line at
     > once. This could easily be converted to pixel scrolling but would be very
     > slow I fear.
     >
     > andy
     > --------------------------------------------------------------
     > Dr Andy Piper
     > Senior Consultant Architect, BEA Systems Ltd




     From:
             Ben Wing <ben@666.com>
                                                          8/10/1999 11:11 PM

      Subject:
             Re: Widgets
          To:
             Andy Piper <andy@xemacs.org>




     I think you might have misinterpreted what i meant.  I meant to say that XEmacs should
     implement the *concept* of a hierarchy of nested child "widgets" or "gui items" or
     whatever we want to call them -- this includes container "widgets" such as grouping
     widgets (which draw a border around the children, like in Windows), tab widgets, simple
     layout widgets (invisible, but lay out their children appropriately), etc, plus leaf
     "widgets" (buttons, sliders, etc., also standard Emacs windows).  The layout calculations
     for these widgets would be handled entirely by XEmacs in a window-system-independent way.
     There is no need to create a corresponding hierarchy of window-system
     widgets/controls/whatever if it's not required, and certainly no need to try to use the
     window-system-supplied geometry management routines.  It's absolutely necessary to support
     this nesting concept in XEmacs, however, or it's impossible to have easily-designable
     dialog boxes.  On the other hand, I think it *is* required to create much of this
     hierarchy within the actual window system, at the very least for non-invisible container
     widgets (tab, grouping, etc.), otherwise we will have very bogus, non-native-looking
     containers like your current tab-widget implementation.  It's critical for XEmacs to be
     able to create dialog boxes in Windows or Motif that look just like those in any other
     standard application.  Otherwise people will continue to think that XEmacs is a
     backwards-looking, badly implemented piece of software, which in many ways it is,
     particularly in regards to its user interface.

     Perhaps we should talk on the phone?  This typing is quite hard for me still.  What hours
     are you at work?  My hours are approx. 2pm - 2am Pacific time (GMT - 7 hours currently).

     ben


     From:
             Ben Wing <ben@666.com>
                                                          7/21/1999 2:44 AM

      Subject:
             Re: Tabs 'n widgets screenshot
          To:
             Andy Piper <andy@xemacs.org>
         CC:
             xemacs-beta@xemacs.org, wmperry@aventail.com




     This is real cool, but looking at this, it's clear that it doesn't look the
     way tab widgets are supposed to work.  In particular, of course, they should
     have the proper borders around the stuff displayed.  I've attached a screen
     shot of a typical Windows dialog box with a tab widget in it.  The problem
     lies with this "expanded gutter" concept.  Tabs are *NOT* extra graphical junk
     placed in the gutters of a buffer but are GUI objects with *children* inside
     of them.  This is the right way to do things, and you would need no extra
     gutter functionality at all for this.  You just need to implement the concept
     of GUI objects containing other GUI objects within them.  One such GUI object
     needs to be a "Emacs-text" GUI object, which is an Emacs window and contains a
     buffer within it.  At this level, you need not be concerned with the
     complexities of geometry layout.  The only change that needs to be made in the
     overall strategy of frames, windows, etc. is that windows need not be exactly
     contiguous and tiled, as long as they are contained within a frame.  Or more
     specifically: Given that you could always split a window contained inside a
     GUI object, we just need to expand things so that each frame has *multiple*
     hierarchies of windows in it, rather than just one.  A hierarchy of windows
     can nest inside of another window -- e.g. I put a tab widget or a text widget
     inside of a buffer.  This should be easy to implement -- just change things so
     there are multiple hierarchies of windows where there are one, each (except
     the top-level one) being rooted inside some other window.

     Anyone willing to implement this? Andy?


     From:
             Ben Wing <ben@666.com>
                                                           6/30/1999 3:30 PM

      Subject:
             Re: Focus Help!
          To:
             Andy Piper <andy@xemacs.org>
         CC:
             Ben Wing <ben@xemacs.org>, martin@xemacs.org, andyp@beasys.com




     It sounds like you're doing very good work.  It also sounds like the approach
     you have followed is the correct one.  Now, it seems like there isn't really
     that much work left to get dialog boxes working.  What you really just need to
     do is implement container widgets, that is to say, subwindows that can contain
     other subwindows.  For example, the tab widget works this way. (It sounds like
     you have already implemented tab widgets, so I don't quite see how you've done
     this without the concept of container widgets.) So you might just try adding a
     framework for container widgets and then implementing very simple container
     widgets.  The basic container widgets are:

     1. A vertical-layout widget, which draws nothing itself and lays out its
     children one above the next.
     2. A horizontal-layout widget, which draws nothing itself and lays out its
     children side-to-side.
     3. A box (or "grouping") widget, which draws a rectangle around its single child
     and optionally draws some text on the top or bottom line of the rectangle.
     4. A tab widget, which displays a series of tabs horizontally at the top of its
     area, and then below it places one of its children,
     corresponding to the selected tab.
     5. A user widget, which draws nothing itself and does no layout at all on its
     children, except that it has a "layout callback"
     property, a Lisp function, so that the programmer can control the layout.

     The framework is as follows:

     1. Every widget has at least the following properties:
          a) a size, whose value can be "unspecified", which might be implemented
     using the value -1.  The default value should be "unspecified".
          b) whether it's mapped, i.e. whether it will be displayed. (Some container
     widgets, such as the tab widget, set the mapped
     property themselves on their children.  Others, such as the vertical and
     horizontal layout widgets, don't change this property but pay attention to it,
     and ignore completely all children marked as unmapped.) The default value should
     be "true".
          c) whether its size can be changed by another widget's layout routine. The
     default value should be "true".
          d) a layout procedure, which (potentially at least) determines the size of
     the widget as well as the position, size and mappedness of its child widgets.
     The layout procedure is inherent in the widget and is not an external property
     of the widget (except in the case of the "user widget"): it is instead more like
     the redisplay callback that each widget has.
     2. Every container widget contains a property which is a list of child widgets.
     3. Every child widget contains the following properties:
          a) a position indicating where the child is located relative to the top
     left corner of its parent.  The position's value can be "unspecified", which
     might be implemented using the value -1.  The default value should be
     "unspecified".
          b) whether its position can be changed by another widget's layout routine.
     The default value should be "true".
     4. All of the properties just listed (except possibly the layout procedure) can
     be modified directly by the programmer, and there are no proscriptions against
     doing so.  However, if the programmer wants to resize, reposition, map or unmap
     a widget in such a way that the layout of all the other widgets in the tree
     changes appropriately, he should use a special function to change the property,
     as described below.

     The redisplay mechanism pays attention to the position, size, and mappedness
     properties and to the hierarchy of widgets, mapping, resizing and repositioning
     the corresponding subwindows (the "real representation" of the widgets) as
     necessary.  It also pays attention to the hierarchy of the widgets, making sure
     that container subwindows get drawn before their child subwindows.  When it
     encounters widgets with an unspecified size, it should not draw them, and should
     issue a warning.  When it encounters widgets with an unspecified position, it
     should draw them at position (0, 0) and should issue a warning.

     The above framework should be fairly simple to implement and is basically
     universal across all high-level windowing system toolkits.  The stickyness comes
     with what procedures you follow for getting the layout done.

     Andy, I understand that implementing this may seem like a daunting task.
     Therefore, I propose that at first you implement the above framework but don't
     implement any of the layout procedures, or any of the functions that call them:
     Just make them stubs that do nothing.  This way, the Lisp programmer can still
     create any dialog boxes he wants, he just has to set the sizes and positions of
     all the widgets explicitly, and then recompute them whenever the widget tree is
     resized (once you get around to allowing this).  I have a lot more to write
     about exactly how the layout procedures work, but I'll send that to you later
     once you're ready.

     You should also think about making a way to have widget trees as top-level
     windows rather than just glyphs in a buffer.  There's already the concept of
     "popup" frames.  You could provide an easy way to create a popup frame with no
     menu, toolbars, scrollbars, modeline or minibuffer, and put a single glyph in
     the displayed buffer that takes up the whole Emacs window.

     Ben




     March 20, 2000

     You wrote to me awhile ago about this and asked about documentation, and I
     dictated a response but never got it sent, so here it is:

     I don't think there's any more documentation on how things work under Xt but it
     should be clear. The EmacsFrame widget is the widget corresponding to the X
     window that Emacs draws into and there is a handler for expose events called
     from Xt which arranges for the invalidated areas to get redrawn. I think this
     used to happen as part of the handler itself but now it is delayed until the
     next call to redisplay.

     However, one thing that you absolutely must not do is remove the Xt support.
     This would be an incredibly unfriendly thing to do as it would prevent people
     from using any widget set other than Qt or GTK. Keep in mind that people run
     XEmacs on all sorts of different versions of X in Unix, and Xt is the standard
     and the only toolkit that probably exists on all of these systems.

     Pardon me if I've misunderstood your intentions w.r.t. this.

     As for how you would implement GTK support, it will not be very hard to convert
     redisplay to draw into a GTK window instead of an Xt window. In fact redisplay
     basically doesn't know about Xt at all, except in the portion that handles
     updating menubars and scrollbars and stuff that's directly related to Xt.

     What you'd probably want to do is create a new set of event routines to replace
     the ones in event-Xt.c. On the display side you could conceivably create a new
     device type but you probably wouldn't want to do that because it would be an
     externally visible change at the Lisp level. You might simply want to put a
     flag on each frame indicating what sort of toolkit the frame was created under
     and put conditions in the redisplay code and the code to update toolbars and
     menubars and so forth to test this flag and do the appropriate thing.


     April 12, 2000

     This is way cool, buuuuutttttttt .............

     what we *really* need is the GUI interface on top of it.  I've taken a shot at
     it with generic-print-buffer
     (print-buffer is taken by lpr, which is such a total mess that it needs to be
     trashed; or at least, the generic
     stuff in this package needs to be taken out and properly genericized).  For
     the moment, generic-print-buffer
     just does something like what Kirill's been posting if we're running windows,
     and uses lpr otherwards.  However, what we absofuckinglutely need is a Lisp
     interface onto `EnumPrinters()' so that we can get the
     list of printers and have a nice menu listing the available printers, and you
     can check the one you want.  People in the Windows world don't normally even
     know the names of their local printers!

     Kirill, given what I've done in `simple.el' and `menubar-items.el', do you think
     you could add the `EnumPrinters()'
     support and fix up the GUI?  If you don't feel comfortable with the GUI, at
     least do the `EnumPrinters()'.

     But ...  Kirill, I tried your formula for printing and nothing happened.
     Perhaps I didn't call redisplay-frame or something?  You need to fix this up
     and make it work for multi-page documents. (Again, this is in
     generic-print-buffer.)  Nothing special, it just needs to fucking work!  There
     are zillions and zillions of postings every day on xemacs-nt about how to get
     printing working, and none seem to refer to the built-in support.

     ben


     April 19, 2000

     Kirill 'Big K' Katsnelson wrote:

     > Some time ago, Ben Wing wrote...
     > >kirill, the interface i created is more general, like this:
     >
     > [snip]
     >
     > >Unfortunately I haven't implemented much of this; just some of the file
     > >dialog box.  but i think
     > >this is better than creating new mswindows-specific primitives.  if you
     > >are interested in working on
     > >this, i'll send you the code i have.
     >
     > Sure. Can you just commit it for my starting point?
     >
     > >also, the dialogs shouldn't have anything directly to do with the printer
     > >device.  all they should
     > >do is return a set of values.  it's the caller's responsibility to
     > >interpret them and set device
     > >properties accordingly.  this way, there's a complete separation between
     > >the underlying
     > >functionality and the gui.
     >
     > Unfortunately. I thought about doing it this way, but we then lose a lot of
     > printer-specific setup in this case. The DEVMODE structure contains two
     > parts: printer independent, as defined by SDK typedef DEVMODE, and
     > some trailing bytes, of unknown structure, used by a driver. The driver
     > only returns the extra length it wants. Such options as PCL ReT resolution
     > enhancement options or PostScript negative output are not available
     > through the standard part of the devmode structure, and stored in the
     > driver part (printer dialogs are driver-specific).
     >
     > So we have total of three options:
     > - Not to implement options beyond standard DEVMODE
     > - Make DEVMODE a Lisp object.
     > - Hide DEVMODE inside the device object.
     >
     > First case looks cheesy. Letting DEVMODE fall off the printer is no good
     > either, since one needs both the device and the devmode to edit the
     > devmode, and they must match. I am still convinced that the devmode and
     > the printer should not be separated.

     hmm, i see ...  this completely breaks abstraction though.  it fails in various
     scenarios, e.g. a program wants to initialize the dialog box with certain
     non-driver-specific properties, without caring about the particular printer.

     i think you should create a new print-properties object that encapsulates all
     printer properties (which can be changed using get/put), including the printer
     name, and contains a DEVMODE in it.  if the printer name gets changed, the
     DEVMODE might change too, but the print-properties object itself stays the
     same.  you pass this object as a parameter to the dialog box, and it gets
     changed accordingly.  you can call something like set-device-print-properties to
     stick everything in this structure into the device. (you could imagine a case
     where someone wanted to keep multiple print configurations around ...)

     >
     >
     > Big K

     --
     Ben


File: internals.info,  Node: Discussion -- Multilingual Issues,  Next: Discussion -- Instantiators and Generic Property Accessors,  Prev: Discussion -- Dialog Boxes,  Up: Future Work Discussion

45.4 Discussion - Multilingual Issues
=====================================


                                                          4/10/2000 4:13 AM

     BTW I am planning on adding some more powerful font-mapping capabilities to
     XEmacs (i.e. how do we map particular characters to the proper fonts that can
     display them, and how do we map the character's codes to the indices into the
     font).  These will replace to hackish charset-registry/charset-ccl-program stuff
     we currently have, and be [a] much more powerful, [b] designed in a
     window-system-independent way, [c] works with specifiers so you can control the
     mapping of individual buffers, and [d] works on a character rather than charset
     level, to correctly handle Unicode.  One possible usage would be to declare that
     all latin1 in a particular buffer to be displayed with latin2 fonts; I bet
     Hrvoje would really appreciate that

     ---------------------------------------------------------------------------

     April 10, 2000

     [info from "creation of generic macros for accessing internally formatted data"]

     Hmm, so there I just wrote a detailed design for the macros.  I would be
     *THRILLED* and overjoyed if you went ahead and implemented this mechanism, or
     parts of it.

     I've just finished arranging for a new transcriptionist, and soon I should be
     able to send off and get back my dictation of my (a) exposing streams to lisp,
     and (b) allowing for proper lisp-created coding systems, which define their
     reading, writing, and detecting methods in lisp.


     BTW How's it going wrt your Unicode and decode-priority stuff?

     And ...  you sent me mail asking what it was you had promised me, and listed
     only one thing, which was
     profiling of vm and certain other operations you found showed tremendous
     slowdown with Japanese characters.  The other main thing I want from you is

     -- Your priorities, as an actual Japanese user and XEmacs developer,
     concerning what MULE work should be done, how it should be done, in what
     order, etc.

     I'm sure there's something else, but it's been awhile since I took my sleeping
     dose and my brain can barely function anymore.  Just let me know how you're
     going to proceed with the above macro changes.

     BTW there's some nice Perl scripts written by Martin and fixed by me to make
     global-search-and-replace
     much, much easier.  I've attached them.  The first one is a shell script that
     works like

     gr foo bar *.[ch]

     and replaces foo with bar in all of the files.  For each modified file, a
     backup is created in the backup/ directory, which is created as necessary.
     This shell script is a fairly trivial front end onto global-replace2, which is
     a perl script that takes one argument (a Perl expression such as s/foo/bar/g)
     and a list of files obtained by reading the stdin, and does the same global
     replacement.  This means that the regexp syntax used here has to be perl-style
     rather than standard emacs/grep style.

     ben

     ---------------------------------------------------------------------


     From:
             Ben Wing <ben@666.com>
                                                         12/23/1999 3:34 AM

      Subject:
             Re: check process state before accessing coding_stream (fix PR#1061)
          To:
             "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>
         CC:
             XEmacs Developers <xemacs-beta@xemacs.org>




     Thankfully, nearly all of this horridity you bring up is irrelevant.  In
     XEmacs, "gettext" does not refer to any standard API, but is merely a stand-in
     for a translation routine (presumably written by us).  We may as well call it
     something else.  We define our own concept of "current language".  We also
     allow for a function that needs a different version for each language, which
     handles all cases where simple translation isn't sufficient, e.g. when you
     have to pluralize some noun given to you or insert the correct form of the
     definite article.  No weird hacks needed.  No interaction problems with other
     pieces of software.

     What I wrote "awhile ago" is (unfortunately) not anywhere public currently,
     but it's on my list to put it on the web site.  "There you go again" is
     usually not true; most of what I quote was indeed put out publicly at some
     point, but I'll try to be more explicit about this in the future.

     ben

     "Stephen J. Turnbull" wrote:

     > >>>>> "Ben" == Ben Wing <ben@666.com> writes:
     >
     >     Ben> "Stephen J. Turnbull" wrote:
     >
     >     >> What I have in mind is not just gettext-izing everything in the
     >     >> XEmacs core sources.  I currently believe that to be
     >     >> unacceptable
     >
     >     Ben> I don't quite understand.  Could you elaborate and give some
     >     Ben> examples?
     >
     > Examples?  Hmm.
     >
     > First, there's the surface of Jan's y-or-n-p example.  You have to
     > coordinate the translation of the message string and the response
     > prompt.  This is handled by y-or-n-p itself (I see that we already do
     > have gettext for Emacs Lisp, that's nice to know).
     >
     > Except that it's not really handled by y-or-n-p.  There's no reason to
     > suppose that somebody writing a Lisp package would necessarily use the
     > XEmacs domain (in fact, due to the way gettext binds text domains---if
     > I understand that correctly---we don't want that to be the case,
     > because it means that every time a Lisp package is updated the whole
     > XEmacs catalog must also be updated).  So which domain gets used for
     > the message string?
     >
     > In the current implementation, it is the domain of y-or-n-p.  So
     > packages with their own domain won't get y-or-n-p prompts correctly
     > translated.  But that means that the package should do its own
     > translation.  But now you're applying gettext to the same string
     > twice; you just have to pray the that translator upstream doesn't
     > collide with an English string that's in the XEmacs domain.  (The
     > gettext docs mention the similar problem of English words with
     > multiple meanings that must map to different words in the target
     > language; this can be disambiguated by various trickeries in forming
     > the strings ... but only if you "own" them, which in the multi-domain,
     > interated gettext example you do not.)  AFAICT this means that you
     > must never pass untranslated strings across public APIs, but this may
     > or may not be reasonable, and certainly is inconvenient.
     >
     > Next, we have to translate the possible answer strings to match the
     > language being passed by the user.  This is presumably OK here,
     > because it's done by y-or-n-p.  But what if y-or-n-p returned a string
     > rather than a boolean?  Then we would need to coordinate the
     > presentation of the prompt (done by y-or-n-p) and the translation of
     > the possible answer strings (done by the caller).  This can in fact be
     > done using dgettext with the XEmacs domain, but you must know that
     > y-or-n-p is in the XEmacs domain.  This is not necessarily going to be
     > obvious, and it might very well be that sets of related packages might
     > have the same domain, so you wouldn't necessarily know which domain is
     > appropriate by looking at the requires.
     >
     > And what happens if one domain does supply translations for a language
     > and the other does not?  AFAIK, gettext has no way to find out if this
     > is the case.  But you might very will prefer a global fallback to
     > English if substantial phrases are drawn from both domains, while you
     > might prefer string-by-string fallback if the main text is translated
     > and only a few words are left to fallback to English.
     >
     > Aside from confusing users, this puts a great burden on programmers.
     > Programmers need to know about the status of the domains of packages
     > they use as well as the XEmacs domain; they need to program
     > defensively against the possibility that some package they use will
     > become gettext-ized, or the translation projects will be out of synch
     > (some teams will do the calling package first, others will do the
     > caller package first).
     >
     > I don't think anybody will use gettext in these circumstances.  At
     > least not after they get the first bug report that "XEmacs is stuck in
     > an infinite y-or-n-p loop and I can't get out."
     >
     >     Ben> I wrote this awhile ago:
     >
     > "There you go again."  Not anywhere I could see it!  (At least, it
     > doesn't look familiar and grepping the archives doesn't turn it up.)
     >
     > OK, you win.  Subscribe me to xemacs-review.  Or whatever seems
     > appropriate.
     >
     > --
     > University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
     > Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
     > _________________  _________________  _________________  _________________
     > What are those straight lines for?  "XEmacs rules."

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it is not apparent in your message,
     please say so.  Thanks for your understanding.



     --------------------------------------------------------------------


     From:
             Ben Wing <ben@666.com>
                                                         12/21/1999 2:22 AM

      Subject:
             Re: check process state before accessing coding_stream (fix PR#1061)
          To:
             "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>
         CC:
             XEmacs Developers <xemacs-beta@xemacs.org>





     "Stephen J. Turnbull" wrote:

     > >>>>> "Ben" == Ben Wing <ben@666.com> writes:
     >
     >     Ben> Implementing message translation is not that hard.
     >
     > What I have in mind is not just gettext-izing everything in the XEmacs
     > core sources.  I currently believe that to be unacceptable (see Jan's
     > message for the pitfalls in I18N; it's worse for M17N).  I think
     > really solving this problem needs a specifier-like fallback mechanism
     > (this would solve Jan's example because you could query the
     > text-specifier presenting the question for the affirmative and
     > negative responses, and the catalog-building mechanism would have
     > checks to make sure they were properly set, perhaps a locale
     > (language) argument), and gettext is just not sufficient for that.

     I don't quite understand.  Could you elaborate and give some examples?

     >
     >
     > At a minimum, we need to implement gettext for Lisp packages.
     > (Currently, gettext is only implemented for C AFAIK.)  But this could
     > potentially cuase more trouble than it's worth.
     >
     >     Ben> A lot depends on priority: How important do you think this
     >     Ben> issue is to your average Japanese/Chinese/etc. user?
     >
     > Which average Japanese (etc) user?  The English-skilled (relatively)
     > programmer in the free software movement, or my not-at-all-competent
     > undergrad students who I would love to have using an Emacs?  This is a
     > really important ease-of-use issue.
     >
     > Realistically, for Japanese, it's low priority.  The Japanese team in
     > the GNU Translation Project is doing very little AFAIK, so even if the
     > capability were there, I doubt the message catalog would soon be done.
     >
     > But I think that many non-English speakers would find it very
     > attractive, and for many languages there are well-organized and
     > productive translation teams.  I suspect that if the I18N facility
     > were well-designed, many Western European languages would have full
     > catalogs within a year (granted, they are the ones where it's least
     > needed :-( ).
     >
     > Personally, I think doing it well is hard, and of little benefit to
     > _current_ core XEmacs constituency.  I think doing a good job, with
     > catalogs, would be very attractive to many non-English-speaking
     > _potential_ users.
     >
     >     Ben> How does it compare to some of the other important Mule
     >     Ben> issues that Martin and I are (trying to work) on?
     >
     > I don't know what you guys are _trying_ to work on.  Everything in the
     > I18N section of "Architecting XEmacs" is red-flagged.  OTOH, it's
     > clear from your posts that you are overburdened, so I can't read
     > priority into the fact that you've responded to specific issues in the
     > past.

     I wrote this awhile ago:


     >
     >     Ben> The big question is, would you be willing to help do the
     >     Ben> actual implementation, to "be my hands"?
     >
     > Sure, subject to the usual caveat that I'd need to be convinced it's
     > worth doing and a secondary caveat that I am not an experienced coder.

     If you'll implement it, I'll design it.  It's more a case of will on your part
     than anything else.  I can give you instructions sufficient enough to match
     your level of expertise.

     ben

     >
     >
     > --
     > University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
     > Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
     > _________________  _________________  _________________  _________________
     > What are those straight lines for?  "XEmacs rules."

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it is not apparent in your message,
     please say so.  Thanks for your understanding.



     -----------------------------------------------------------------------------

     Dec 20, 1999


     Implementing message translation is not that hard.  I've already done a lot of
     preliminary work in places such as `make-msgfile.lex' in lib-src/.  Finishing up
     the work is not that big a task; I already know exactly how it should be
     done.  Perhaps I'll write up detailed design instructions for this, as I'm
     doing for other things.  A lot depends on priority: How important do you think
     this issue is to your average Japanese/Chinese/etc. user?  How does it compare
     to some of the other important Mule issues that Martin and I are (trying to
     work) on?  If I did the design document, would you be willing to do the
     necessary bit of C hackery to implement the document?  If the design document
     is not specific enough for you, I can give you an "implementation document"
     which will definitely be specific enough: i.e. I'll show you exactly where the
     code needs to be modified, and how.  The big question is, would you be willing
     to help do the actual implementation, to "be my hands"?

     ---------------------------------------------------------------------------

     From:
             Ben Wing <ben@666.com>
                                                         12/14/1999 11:00 PM

      Subject:
             Re: Mule UI disaster: displaying character tables
          To:
             Hrvoje Niksic <hniksic@iskon.hr>
         CC:
             XEmacs vs Mule <xemacs-mule@xemacs.org>




     What I mean is, please put my name in the header, as well as xemacs-mule.
     That way I'll see it in my personal box.

     I agree that Mule has problems, but:

     Brokenness can be fixed.
     Slowness can be fixed.
     Limitations can be fixed.

     The design limitation you mention below, for example, is not really very
     hard to change.

     Keep in mind that I pretty much rewrote Mule from scratch, and did it
     *all* in 6-7 months.  In comparison with that, the changes below are
     pretty minor, and each could be done by a good (and able-bodied!)
     programmer familiar with the Mule code in less than a week -- to the
     XEmacs code, at least.  The problem is, everyone who could do this work is
     instead spending their time complaining about Mule problems instead of
     doing things.

     I'll gladly help out anyone who wants to do Mule coding by explaining all
     the details; I'll even write a "Mule internals manual", if that will
     help.  I can also make international phone calls -- they're cheap here in
     the US due to the long distance wars.  But so far no one has asked me for
     help or shown any willingness to do any work on Mule.

     Perhaps people are daunted by the seeming vastness of the problems.  But I
     wager that if I had another 6 months to work on nothing but Mule, it would
     be nearly perfect.  The basic design of the XEmacs C code is good;
     incremental changes, without over-much concern for compatibility, could
     make huge strides in a short amount of time (as was the case the whole
     time I worked on it, esp. towards the end -- it didn't even *compile* for
     4 months!).  A "total rewrite" would be an incredible waste of time.

     Again, I'm completely willing to provide help, documentation, design
     improvement suggestions (ala Architecting XEmacs -- which seems to have
     been completely ignored, alas), etc.

     ben

     Hrvoje Niksic wrote:

     > Ben Wing <ben@666.com> writes:
     >
     > > I'm the one who did most of the Mule work in XEmacs, so if you have
     > > any questions about the core, please address them to me directly.  I
     > > can probably give you a very clear and detailed answer.
     >
     > Thanks.  I think it still makes sense to ask here, so that other
     > developer have a chance to chime in.
     >
     > > However, I need some explanation.  What's misdesigned that you're
     > > complaining about?  And what's the coding-system disaster?
     >
     > It's been spoken of a lot.  Basically:
     >
     > * Unlike XEmacs/no-Mule, XEmacs/Mule doesn't preserve binary files in
     >   Latin 2 locales by default.  This is annoying for users who are used
     >   to XEmacs/no-Mule.
     >
     > * XEmacs/Mule is much slower than XEmacs, and not only because of
     >   character/byte conversions.  It seems that font lookups etc. are
     >   slower.
     >
     > * The "coding-system disaster" refers to inherent limitations of the
     >   coding-system model.  If I understand things correctly,
     >   coding-systems convert streams of bytes to streams of Emchars.  It
     >   does not appear to be possible to create a "gzip" coding system for
     >   handling gzipped file.  Even EOL conversions look kludgish:
     >
     >     iso-2022-8
     >     iso-2022-8-dos
     >     iso-2022-8-mac
     >     iso-2022-8-unix
     >     iso-2022-8bit-ss2
     >     iso-2022-8bit-ss2-dos
     >     iso-2022-8bit-ss2-mac
     >     iso-2022-8bit-ss2-unix
     >     iso-2022-int-1
     >     iso-2022-int-1-dos
     >     iso-2022-int-1-mac
     >     iso-2022-int-1-unix
     >
     >   Ideally, it should be possible to specify a stream of
     >   coding-systems, where only the last one converts to actual Emchars.
     >
     > There are more problems I don't remember right now.  Many many usage
     > problems become apparent when I stand and look over the shoulders of
     > an XEmacs users who tries to use Mule.

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it is not apparent in your message,
     please say so.  Thanks for your understanding.



     -----------------------------------------------------------------------




     From:
             Ben Wing <ben@666.com>
                                                        12/14/1999 12:20 AM

      Subject:
             Re: Mule UI disaster: displaying character tables
          To:
             "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>
         CC:
             XEmacs vs Mule <xemacs-mule@xemacs.org>




     I think you should go ahead with your proposal, and assume it will get
     implemented.  I don't think Martin is really suggesting that API changes not
     be allowed, but just that they proceed in a somewhat orderly fashion; and in
     any case, I imagine I have final say in cases of Mule-related conflicts.

     ben

     "Stephen J. Turnbull" wrote:

     > >>>>> "Hrvoje" == Hrvoje Niksic <hniksic@iskon.hr> writes:
     >
     >     Hrvoje> So next I tried the "Mule" menu.  That's right, boys and
     >     Hrvoje> girls, I've never looked at it before.
     >
     > For quite a while, it didn't work at all, led to crashes and other
     > warm/fuzzy things.  IIRC there used to be a top level menu item
     > pointing to information about the current language environment but it
     > got removed.
     >
     >     Hrvoje> Wow.  Seeing shift_jis, iso-2022 variants and (above all
     >     Hrvoje> things) big5 makes me really warm and fuzzy.
     >
     > We've been through this recently---you were there.  We know what to do
     > about it, basically (Ben liked my proposal, and it would fix this
     > silliness as well as the binary file breakage).  But given that Ben
     > and Martin seem to have different ideas about where to go with Mule
     > (Ben seemed to be supporting API and implementation revisions, Martin
     > evidently wants to keep the current Mule), working on that proposal is
     > possibly a waste of time.  I've got other stuff on my plate and I'll
     > get back to it one of these days (not tomorrow but sooner than Real
     > Soon Now).
     >
     >     Hrvoje> The items it presents (leading to further submenus) are:
     >
     >     Hrvoje>     94 character set
     >     Hrvoje>     94 x 94 character set
     >     Hrvoje>     96 character set
     >
     > This _is_ bad UI, now that you point it out.  But it is quite natural
     > for a coding system lawyer (as all Japanese users have to be), I never
     > noticed it before.  Easy enough to fix ("raise my karma").
     >
     >     Hrvoje> But I do bear some Mule scars, so I happily select "96
     >     Hrvoje> character sets", then ISO8859-2.  And I get this:
     >
     > [Table omitted]
     >
     >     Hrvoje> So me wonders: what the hell is this?
     >
     > Huh?  That is the standard table that you see over and over again in
     > references.  I'll believe you if you say you've never seen one before,
     > but every Japanese users' manual has dozens of pages of those, using
     > exactly that format.
     >
     > The presentation in the range 00--7F is not unreasonable for Latin 2;
     > ISO-8859 is a version of ISO-2022, so the high bit should not be
     > interpreted as "+ x80" (technically speaking), it should be
     > interpreted as a character set shift.
     >
     > Of course, this doesn't make sense to anybody but a character set
     > lawyer, and so should be changed.  Especially since the header refers
     > to ISO-8859-2 which everybody these days thinks of as _one, 8-bit_
     > character set, not two 7-bit ones.
     >
     > As for the "Japanese" in the table, that's just a really stupid
     > "optimization": those happen to be line-drawing characters available
     > in JIS X 0208, to make pretty borders.  Substitute "-", "+", and "|"
     > in appropriate places to make ugly but portable borders.
     >
     >     Hrvoje> Mule is just broken.  Warn your friends.
     >
     > Hrvoje is on the rampage again.  Warn your friends ;-)
     >
     > --
     > University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
     > Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
     > _________________  _________________  _________________  _________________
     > What are those straight lines for?  "XEmacs rules."

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it is not apparent in your message,
     please say so.  Thanks for your understanding.



     ---------------------------------------------------------------------------

     From:
             Ben Wing <ben@666.com>
                                                         12/14/1999 10:28 PM

      Subject:
             Re: Autodetect proposal; specifer questions/suggestions
          To:
             "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>




     I've always thought the specifier API is too complicated (and too
     "write-only"), but I went back at one point well after I designed it and I
     couldn't figure out an obvious way to simplify it that still kept reasonable
     functionality.  Perhaps that's what Custom did, and why it turned out bad.

     Inefficiency is a stupid reason not to use them.  They seem efficient enough
     for redisplay.  Changing them might be inefficient, but Emacs Lisp is in
     general, right?

     Can you propose an API or functionality change that will make them more used?



     "Stephen J. Turnbull" wrote:

     > >>>>> "Ben" == Ben Wing <ben@666.com> writes:
     >
     >     Ben> I think you should go ahead with your proposal, and assume it
     >     Ben> will get implemented.
     >
     > OK.  "yas baas" ;-)
     >
     > On something totally different.  I'm really bothered by the fact that
     > specifiers are so little used (eg, Custom reimplements them badly),
     > and the fact that every package seems to define its own set of faces
     > (or whatever), rather than use the specifier mechanism to inherit from
     > existing ones, or add new specifications to existing ones.  API problem?
     >
     > Also, faces (maybe specifiers in general?) should have an autoload
     > mechanism, and a `<package>-faces.el' (or `<package>-specifiers.el')
     > convention.  There are a number of faces in (eg) Custom that I like to
     > use, but I have to load Custom to get them.  And Custom should be able
     > to somehow see all the faces in various packages available, even when
     > they are not loaded.
     >
     > I've seen claims that specifiers aren't very efficient.
     >
     > Opinions?
     >
     > --
     > University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
     > Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
     > _________________  _________________  _________________  _________________
     > What are those straight lines for?  "XEmacs rules."

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it is not apparent in your message,
     please say so.  Thanks for your understanding.


     -----------------------------------------------------------------------------
     From:
             Ben Wing <ben@666.com>
                                                          11/18/1999 9:02 PM

      Subject:
             Re: Char-related crashes (hopefully) fixed
          To:
             "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>
         CC:
             XEmacs Beta List <xemacs-beta@xemacs.org>




     OK, in summation:

     1. C-q is a user-level function and should do whatever makes the most sense.
     2. int-char is a low-level primitive and should never depend on high-level
     settings like language environment.
     3. Everything you can do with int-char can and should be done with make-char
     -- representation-independent, much less likelihood of bugs, etc.  Therefore
     int-char should be removed.
     4. Note that CLTL2 also removes int-char.
     5. Your statement

     > In one-byte buffers (either Olivier's 1/2/4 extension or `xemacs -font
     > *-iso8859-2') it implicitly will have dependence whatever you say.

     is confusing internal and external representations.

     ben

     "Stephen J. Turnbull" wrote:

     > Can somebody give a bunch of examples where using integers as
     > characters is useful?  For that matter, where they are actually used?
     > Ben said "backward compatibility," but I haven't seen this used, and I
     > don't really know how to grep for it.  I have grepped for int-char,
     > int-to-char, char-int, and char-to-int and they're pretty rare in the
     > core and package code (2/3 of it) that I have.
     >
     > The only one that I ever use is the C-q hack for inserting characters
     > by code value at the keyboard, and that could arguably (and in
     > Japanese invariably is) delegated to an input method which would know
     > about language environment (and return a true character).
     >
     > For iterating over a character set in "natural" order, only ASCII
     > satisfies the requirement of having one, and even that's shaky.  AFAIK
     > the Swedes and the Norwegians, or is it the Danes, disagree on
     > ordering the _letters_ in ISO-8859-1 character set.  This really
     > should be table-driven, and will have to be for everything except
     > ASCII and ISO-8859-1 if we go to a Unicode internal representation.
     >
     > We already have primitives for efficient case conversion and the like.
     >
     > The only example I can think of offhand where you would really really
     > want the facility is to iterate over a code space where you don't know
     > which points are legal characters.  Eg, to print out tables of fonts.
     > Pretty specialized.  And this can be done through make-char, anyway.
     >
     > According to CLtL1, the main portable use for char-int is for hashing.
     > But that doesn't square with the kind of usage we've been talking
     > about (in loops and the like).
     >
     > What else am I missing?
     >
     > Ben's desiderata have some problems.
     >
     > >>>>> "Ben" == Ben Wing <ben@666.com> writes:
     >
     >     Ben> Either int-char should be the mirror opposite of char-int
     >     Ben> (i.e. accept all legal char integers), or it should be
     >     Ben> removed entirely.
     >
     > OK.  I agree with this.
     >
     >     Ben> int-char should *never* have any dependence on the language
     >     Ben> environment.
     >
     > In one-byte buffers (either Olivier's 1/2/4 extension or `xemacs -font
     > *-iso8859-2') it implicitly will have dependence whatever you say.
     > Even without Mule, people can always use external encoders to change
     > raw ISO-8859-2 to ISO-2022 (not that anybody sane ever would, OK,
     > Hrvoje?).  Then the two files will be interpreted differently in a
     > Latin-1 locale Mule; the ISO-8859-2 file will be recognized as
     > ISO-8859-1, and the ISO-2022 file will be internally interpreted as
     > ISO-8859-2.
     >
     > The point is that people normally assume that int-char should accept
     > their "natural" integer to character map.  For Americans, that's
     > ASCII, for Germans, that's ISO-8859-1, for Croatians, that's
     > ISO-8859-2.  And it works "correctly" in a no-mule XEmacs with `-font
     > *-iso8859-2'!  Japanese usually use ku-ten or JIS, and there's a
     > "natural" map from byte-sized integer pairs to shorts, but it's full
     > of holes.  So language environments don't agree on what a legal char
     > integer is, and where they do (eg, ISO-8859-1 and ISO-8859-2), they
     > don't agree on the map.  To satisfy your dictum (with which I agree,
     > but I take to mean we should get rid of these functions) we can take
     > the intersection where they agree
     >
     > ==> legal char integers == ASCII
     >
     > which is what I prefer, or pick something arbitrary and efficient
     >
     > ==> char-int returns the internal representation
     >
     > which I really hate, or something else.  Suggestions?
     >
     >     Ben> I don't think C-q should either.  If Hrvoje wants to insert
     >     Ben> Latin-2 characters by number, then make C-u C-q work so that
     >     Ben> it also prompts for a character set, with a default chosen
     >     Ben> from the language environment.
     >
     > And restrict this to ASCII?  Or assume Latin-1 in GR if there is no
     > prefix argument?
     >
     > This is a useful feature.  C-q currently inserts Latin-2 characters
     > for Hrvoje in no-mule XEmacs (stretching the point only a little); I
     > think it should continue to do so in Mule.  This really is an input
     > method issue, not a keyboard issue.  In XEmacs, inserting an integer
     > into a buffer has no meaning.  Users insert characters.  So this is a
     > completely different issue from the programming API, and should not be
     > considered analogous.
     >
     > Maybe we could have C-q insert according to the Unicode standard, and
     > treat C-u C-q as part of the input method.  But I think most users
     > would prefer to have C-q insert according to their locale-standard
     > tables, and select Unicode explicitly using the C-u C-q idiom.  In
     > fact (again this points to the input method idea), Japanese users
     > would probably like to have the alternatives of using kuten (pairs
     > from 1--94 x 1--94) or JIS (pairs from 0x21--0x7E x 0x21--0x7E) as
     > options since both indexing systems are common in tables.
     >
     > --
     > University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
     > Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
     > __________________________________________________________________________
     > __________________________________________________________________________
     > What are those two straight lines for?  "Free software rules."

     --
     ben

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it's not apparent in your message,
     please say so.  Thanks for your understanding.



     -----------------------------------------------------------------------------

     From:
             Ben Wing <ben@666.com>
                                                         11/16/1999 11:03 PM

      Subject:
             Re: Char-related crashes (hopefully) fixed
          To:
             Yoshiki Hayashi <t90553@m.ecc.u-tokyo.ac.jp>
         CC:
             Hrvoje Niksic <hniksic@iskon.hr>,
             XEmacs Beta List <xemacs-beta@xemacs.org>




     Either int-char should be the mirror opposite of char-int (i.e. accept all
     legal char integers), or it should be removed entirely.

     int-char should *never* have any dependence on the language environment.

     I don't think C-q should either.  If Hrvoje wants to insert Latin-2
     characters by number, then make C-u C-q work so that it also prompts for a
     character set, with a default chosen from the language environment.

     ben

     Yoshiki Hayashi wrote:

     > Hrvoje Niksic <hniksic@iskon.hr> writes:
     >
     > > As Ben said, now that we've fixed the actual bugs, we can think about
     > > changing the behaviour for int-char conversions for 21.2.
     >
     > Following are proposed which integers should be accepted
     > where characters are expected:
     >
     > 1) Don't allow anything
     > 2) Accept 0-127
     > 3) Accept 0-256
     > 4) Accept everything
     >
     > Other things proposed are:
     >
     > a) When doing C-q, treat 128-256 as Latin-2 in Latin 2
     >    language environment.
     >
     > So far, most of the proposal is intended to apply to every
     > int-char conversions, I'd like to make some functions to
     > accept.
     >
     > My plan is:
     > Accept only 0-256 in every place except int-to-char.
     > int-to-char accepts every valid integers.
     > Make new function which does int-to-char conversion
     > correctly according to the language environment.
     >
     > This way, most of the code which does (insert (1+ ?a)) or
     > something continues working. Now internal representation is
     > changed a little bit, so disabling > 256 characters will
     > warn those who are dealing with internal representation
     > directly, which is bad. Still, you can do
     > (let ((i 1442))
     >   (while (i < 2000)
     >     (insert (int-to-char i))
     >     (setq i (+1 i))))
     > to achieve old behaviour.
     >
     > For C-q, I'm not for changing it's original definition,
     > since it might confuse people who are expecting Latin-1 in
     > other language environment and typing just 1 integer doesn't
     > make sense for multibyte world. It's cleaner to make new
     > function, which does make-char according to the charset of
     > language-info-alist so that people who use that often can
     > bind it to C-q or some other keys.
     >
     > --
     > Yoshiki Hayashi

     --
     ben

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it's not apparent in your message,
     please say so.  Thanks for your understanding.


File: internals.info,  Node: Discussion -- Instantiators and Generic Property Accessors,  Next: Discussion -- Switching to C++,  Prev: Discussion -- Multilingual Issues,  Up: Future Work Discussion

45.5 Discussion - Instantiators and Generic Property Accessors
==============================================================

From: Ben Wing <ben@666.com> Date: Sun, 05 May 2002 05:40:07 -0700
Subject: generic functions, new instantiator API

   I've been reading the C++ manual and getting polymorphism,
inheritance, generic functions, etc. in my head.

   We have our own "generic function" already in terms of `get', `put',
etc. which accept various objects.  i'm thinking of extending them so
they can accept, as well as objects, lists (either alists or plists) or
plist-style vectors, and manipulate their properties.  what do people
think of this?

   Also, i'm designing a new API for "instantiators", which are objects
whose main purpose is to hold properties and provide a way of notifying
their containing specifiers when they change.  Instantiator objects are
used when the instantiator gets sufficiently complicated that using
lists and vectors gets unwieldy - e.g. when creating widget trees, such
as would appear in dialog boxes.  you want the ability to
programmatically traipse up and down the tree and dynamically modify a
part of the tree - e.g. a property on a single widget - as necessary,
and have the internal code automatically notice this change and performs
any necessary updates.  lists and vectors are too low-level for this -
no way to get their parent, no way for internal code to be notified when
changes occur, can't always maintain object identity when making
property changes, no way to error-check illegal changes, etc.

   You could also extend this api to cover toolbars; it would probably
make toolbar manipulation significantly easier.  but you'd have to think
about backward compatibility in such cases.

   here is what the api looks like so far - making use of a newly-added
facility for keyword args in primitives.  comments are welcome.

     DEFUN ("make-instantiator", Fmake_instantiator, 1, MANY, 0, /*
     Create a new instantiator object from TYPE and PROPS.
     TYPE should be one of the image instantiator formats described in
     `make-glyph'.
     The rest of the arguments should be keyword properties and associated
     values,
     as also described in `make-glyph'.

     TYPE can also be an old-style vector instantiator.

     Instantiator objects can be used as instantiators (see `make-specifier') in
     glyphs in place of old-style vector instantiators.  They are especially
     used for complicated, nested graphical elements such as widgets (buttons,
     text fields, etc.) -- in fact, widget instantiators will automatically be
     converted into instantiator objects if they are given in vector format.

     Individual properties on instantiators can be manipulated using
     `set-instantiator-property'.  If the property's value is a list (for
     example, a list of children), you can also use `add-instantiator-item'
     to add or insert individual elements in the list.

     `delete-instantiator-item' can be used to delete individual items in the
     list;
     `get-instantiator-item' to locate individual items in the list; and
     `get-instantiator-item-position' to return the position of individual
     items in
     the list.

     `map-instantiator' can be used to (recursively or not) map over an
     instantiator and its children.

     `find-instantiator' can be used to (recursively or not) locate an
     instantiator
     in a tree composed of an instantiator and its descendants.
     */
            /* (type &rest props) */
            (int nargs, Lisp_Object *args))
     {
       /* ^^#### */ return Qnil;
     }

     DEFUN ("set-instantiator-property", Fset_instantiator_property, 3, 3, 0, /*
     Set property PROP to VALUE in INSTANTIATOR.
     INSTANTIATOR should have been created with `make-instantiator'.
     Valid properties depend on the instantiator type and are described in
     `make-glyph'.  For properties that are lists of items, individual items
     can be added or deleted using `add-instantiator-item' and
     `delete-instantiator-item'.

     For compatibility, this also accepts an old-style vector instantiator, and
     destructively modifies it; in this case, adding a property requires
     creating a new vector, which is returned.  You need to use
     `set-glyph-image' on glyphs, or `set-specifier-dirty-flag' on the result of
     `glyph-image', to register instantiator changes to vector
     instantiators. (New-style instantiators automatically convey property
     changes to any glyphs they have been attached to.)
     */
            (instantiator, prop, value))
     {
       Lisp_Object *elt;
       int len;

       /* ^^#### */
       CHECK_VECTOR (instantiator);
       if (!KEYWORDP (prop))
         invalid_argument ("instantiator property must be a keyword", prop);

       elt = XVECTOR_DATA (instantiator);
       len = XVECTOR_LENGTH (instantiator);

       for (len -= 2; len >= 1; len -= 2)
         {
           if (EQ (elt[len], prop))
         {
           elt[len + 1] = value;
           break;
         }
         }

       /* Didn't find it so add it. */
       if (len < 1)
       {
         Lisp_Object alist = Qnil, result;
         struct gcpro gcpro1;

         GCPRO1 (alist);
         alist = tagged_vector_to_alist (instantiator);
         alist = Fcons (Fcons (prop, value), alist);
         result = alist_to_tagged_vector (elt[0], alist);
         free_alist (alist);
         RETURN_UNGCPRO (result);
       }

       return instantiator;
     }

     DEFUN ("instantiator-property", Finstantiator_property, 2, 3, 0, /*
     Return the property PROP of INSTANTIATOR, or DEFAULT if PROP has no value.
     INSTANTIATOR should have been created with `make-instantiator'.
     */
            (instantiator, prop, default_))
     {
       /* ^^#### */ return Qnil;
     }

     DEFUN ("instantiator-properties", Finstantiator_properties, 1, 1, 0, /*
     Return a plist of all defined properties in INSTANTIATOR.
     INSTANTIATOR should have been created with `make-instantiator'.
     */
            (instantiator))
     {
       /* ^^#### */ return Qnil;
     }

     DEFUN ("instantiator-type", Finstantiator_type, 1, 1, 0, /*
     Return the type of INSTANTIATOR.
     INSTANTIATOR should have been created with `make-instantiator'.
     Valid types are the instantiator formats described in `make-glyph'.
     */
            (instantiator))
     {
       /* ^^#### */ return Qnil;
     }

     DEFUN ("instantiator-parent", Finstantiator_parent, 1, 1, 0, /*
     Return the parent of INSTANTIATOR.
     INSTANTIATOR should have been created with `make-instantiator'.
     */
            (instantiator))
     {
       /* ^^#### */ return Qnil;
     }

     DEFUN_WITH_KEYWORDS ("map-instantiator", Fmap_instantiator, 2, 2, 1, 0,
     0, /*
     Map FUN recursively over INSTANTIATOR and its descendants.
     FUN is called with one argument, the INSTANTIATOR.
     If:norecurse is non-nil, don't recurse, just map over the direct
     children (not including the instantiator itself).
     */
                  (fun, instantiator),
                  (norecurse))
     {
       /* ^^#### */ return Qnil;
     }

     DEFUN_WITH_KEYWORDS ("find-instantiator", Ffind_instantiator, 3, 3,
                  1, 0, 0, /*
     Find an instantiator by PROP and VALUE in INSTANTIATOR and its descendants.
     Returns first item which has PROP set to VALUE.
     If:norecurse is non-nil, don't recurse, just look through the direct
     children (not including the instantiator itself).
     */
                  (instantiator, prop, value),
                  (norecurse))
     {
       /* ^^#### */ return Qnil;
     }

     DEFUN_WITH_KEYWORDS ("add-instantiator-item", Fadd_instantiator_item, 3,
     3, 7,
                  0, 0, /*
     Add an item to an instantiator property that's a list of items.
     \(E.g. the children of an instantiator).  PROP is the property whose list of
     items is being modified, and ITEM is the item to add.  To insert somewhere
     before the end, use one of the keywords:

     --:position specifies a zero-based index of an item, and the new item
     will be
     inserted just before the item indicated by the position.  Negative numbers
     count from the end -- thus -1 will cause insertion before the last item, -2
     before the second-to-last item, etc.

     --:before-item and :after-item specify items to insert before or after.
     :test (defaults to `eq') can be used to specify the way to compare the given
     item with existing items.

     --:before-property and :after-property search for an item to insert
     before or
     after by looking for an item with the given property.  If :value is
     given, the
     property must have that value; otherwise, it simply must exist.  This method
     of insertion works if the items in PROP's list are anything that can have or
     hold properties.  \("To have and to hold, for ever and ever ...")  This
     includes:

     -- any object for which `get' works
     -- else, if object is a vector, assume it's a plist-style vector
     -- else, if object is a cons, and its first element is also a cons,
        assume it's an alist
     -- else, if object is a cons, assume it's a plist
     */
                  (instantiator, prop, item),
                  (position, before_item, after_item, test,
                   before_property, after_property, value))
     {
       /* ^^#### */ return Qnil;
     }

     DEFUN_WITH_KEYWORDS ("delete-instantiator-item", Fdelete_instantiator_item,
                  2, 2, 5, 0, 0, /*
     Delete an item in an instantiator property that's a list of items.

     \(E.g. the children of an instantiator).  PROP is the property whose list is
     being searched.  One of these keywords should be given:

     --:position specifies a zero-based index of an item.  Negative numbers
     count from the end -- thus -1 will cause insertion before the last item, -2
     before the second-to-last item, etc.

     --:item specifies the item to delete. :test (defaults to `eq') can be
     used to
     specify the way to compare the given item with existing items.

     --:property searches for an item with the given property.  If :value is
     given, the property must have that value; otherwise, it simply must exist.
     This method of insertion works if the items in PROP's list are anything that
     can have or hold properties -- see `add-instantiator-item'.
     */
                  (instantiator, prop),
                  (item, test, position, property, value))
     {
       /* ^^#### */ return Qnil;
     }

     DEFUN_WITH_KEYWORDS ("get-instantiator-item", Fget_instantiator_item,
                  2, 2, 3, 0, 0, /*
     Get an item in an instantiator property that's a list of items.

     \(E.g. the children of an instantiator).  PROP is the property whose list is
     being searched.  One of these keywords should be given:

     --:position specifies a zero-based index of an item.  Negative numbers
     count
     from the end -- thus -1 will cause insertion before the last item, -2 before
     the second-to-last item, etc.

     --:property searches for an item with the given property.  If :value is
     given, the property must have that value; otherwise, it simply must exist.
     This method of insertion works if the items in PROP's list are anything that
     can have or hold properties -- see `add-instantiator-item'.
     */
                  (instantiator, prop),
                  (position, property, value))
     {
       /* ^^#### */ return Qnil;
     }

     DEFUN_WITH_KEYWORDS ("get-instantiator-item-position",
                  Fget_instantiator_item_position,
                  2, 2, 4, 0, 0, /*
     Return an item's position in an instantiator property that's a list of
     items.

     \(E.g. the children of an instantiator).  PROP is the property whose list is
     being searched.  One of these keywords should be given:

     --:item specifies the item to search for. :test (defaults to `eq') can be
     used to specify the way to compare the given item with existing items.

     --:property searches for an item with the given property.  If :value is
     given, the property must have that value; otherwise, it simply must exist.
     This method of insertion works if the items in PROP's list are anything that
     can have or hold properties -- see `add-instantiator-item'.
     */
                  (instantiator, prop),
                  (item, test, property, value))
     {
       /* ^^#### */ return Qnil;
     }

     DEFUN ("image-instance-instantiator", Fimage_instance_instantiator, 1,
     1, 0, /*
     Return the instantiator from which IMAGE-INSTANCE was created.
     */
            (image_instance))
     {
       /* ^^#### */ return Qnil;
     }

   some other useful stuff:

     DEFUN ("make-image-instance", Fmake_image_instance, 1, 4, 0, /*
     Return a new `image-instance' object.

     Image-instance objects encapsulate the way a particular glyph (pixmap,
     widget, etc.) is displayed on a particular device.  In most circumstances,
     you do not need to directly create image instances; instead, you create a
     glyph using `make-glyph' and add settings (or "instantiators") onto it
     using `set-glyph-image', and XEmacs creates the image instances as
     necessary.  However, it may occasionally be useful to explicitly create
     image instances, if you want more control over the instantiation process.

     For more information on instantiators and instances, see `make-specifier'.

     DATA is an image instantiator, which describes the image; see `make-glyph'
     for a description of the allowed values.

     The most likely circumstance where you need to deal directly with image
     instances is in widget callbacks -- e.g. the callback that's executed when
     a button is pressed in a dialog box of type `general' (see
     `make-dialog-box').  In this case, the widget that was activated is
     described by an image instance. (The callback is usually be written as an
     interactive function with an interactive spec of (interactive \"e\"), and a
     single `event' argument.  The event will be an activate event, describing
     the user action that trigged the callback.  The image instance is
     retrievable from the event using `event-image-instance'.  Handling the
     action may involve setting properties on the image instance or other image
     instances in the dialog box in which the widget is usually contained -- or
     changing the instantiator that generated the image instance, if you want
     permanent changes that will be reflected the next time the dialog box is
     popped up.  Properties on an image instance are set using
     `set-image-instance-property'.  If the widget is part of a hierarchy of
     widgets (as is usually the case in a dialog box, but may not apply if the
     widget was inserted by itself in a buffer [by creating a glyph and
     attaching it to an extent -- see `make-glyph']), there will be a
     corresponding hierarchy of image instances to describe this particular
     instance of the dialog box.  You can retrieve other image instances in the
     hierarchy using primitives such as `image-instance-parent',
     `image-instance-children', and `find-image-instance'.

   ...

     (defun image-instance-property (image-instance property &optional default)
       "Return the given property of the given image instance.
     Returns DEFAULT if the property or the property method do not exist for
     the image instance in the domain."
       (check-argument-type 'image-instance-p image-instance)
       (get image-instance property default))

     (defun set-image-instance-property (image-instance prop value)
       "Set the property PROP on IMAGE-INSTANCE to VALUE.
     Only certain properties of the image instance can be changed, and they
     represent \"temporary\" changes.  If you want to make permanent changes,
     you need to change the instantiator that generated the instance --
     retrieve the instantiator with `image-instance-instantiator', and change
     its properties with `set-instantiator-property'.

     This applies mostly to widgets.  For example, you can set a property on
     a widget image instance to change the state of a radio or checkbox button,
     set the text currently in an edit field, etc.  However, those changes apply
     only to the *currently* displayed widgets.  If these widgets are in a dialog
     box, and you want to change the way the widgets in the dialog box appear
     *each* time the dialog box is displayed, you need to change the
     instantiator.

     Make sure you understand the difference between instantiators and
     instances.  An \"instantiator\" is a specification, indicating how to
     determine the value of a setting whose value can vary in different
     circumstances or \"locales\" (buffers, frames, etc.).  An \"instance\"
     is the
     resulting value in a particular circumstance.  For more information, see
     `make-specifier'."
       (check-argument-type 'image-instance-p image-instance)
       (put image-instance prop value))

   From: "Stephen J. Turnbull" <stephen@xemacs.org> Date: 06 May 2002
16:40:46 +0900

   >>>>> "Ben" == Ben Wing <ben@666.com> writes:

   Ben> We have our own "generic function" already in terms of `get',
 Ben> `put', etc. which accept various objects.

   I proposed extending the class to stuff like charsets about two years
ago, and I think you were one of the folks who objected.

   Ben> i'm thinking of extending them so they can accept lists
Ben> (either alists or plists) or plist-style vectors, and     Ben>
manipulate their properties.  what do people think of this?

   I think extending to lists is something we should approach
cautiously.  For one thing, if "get" is polymorphic, "put" would have
to be too.  But how does it decide when dealing with "nil"?

   Ben> you want the ability to programmatically traipse up and down
Ben> the tree and dynamically modify a part of the tree - e.g. a
Ben> property on a single widget - as necessary, and have the     Ben>
internal code automatically notice this change and performs     Ben>
any necessary updates.

   I like this.

   From: "Stephen J. Turnbull" <stephen@xemacs.org> Date: 07 May 2002
11:17:05 +0900

   >>>>> "Neal" == Neal D Becker <nbecker@hns.com> writes:

   Neal> I thought that generic polymorphism was inherent in lisp, as
 Neal> it is dynamically evaluated.  Why would you need anything
Neal> special in the way functions are written to support generic
Neal> programming?

   I think it's basically a technical matter.  We have a number of
objects that have property lists besides symbols.  Many of them have
special functions (coding-system-get, coding-system-property,
charset-property are examples I find particularly obnoxious).  I would
like to make these obsolete by allowing `get' on charsets, coding
systems, etc.

   And currently we have

   (let ((p (symbol-plist symbol))) (plist-get p prop))

   Ben would like to allow

   (let ((p (symbol-plist symbol))) (get p prop))

   with `get' determining whether P is a plist or an alist.  And where
Michael says "why not use hash tables?", I see `(get hash key)'
(probably to Michael's horror ;-).

   This isn't Lisp any more, though, in some sense.  But then we haven't
been that for years.  AFAIK all real Lisps restrict `get' to symbols.

   From: sperber@informatik.uni-tuebingen.de (Michael Sperber [Mr.
Preprocessor]) Date: Tue, 07 May 2002 08:52:53 +0200

   Indeed.  I'll just say "goosebumps."  But I don't see why it has to
be GET that accesses the plist.  You just build more dispatch into GET
with no immediate benefit to the API.  Ad-hoc genericity gets you
something when there's some place in the code you don't know what the
underlying object is.  I don't see this being the case here.  Why do
you find them "obnoxious?"

   Stephen> This isn't Lisp any more, though, in some sense.  But then
we haven't Stephen> been that for years.  AFAIK all real Lisps restrict
`get' to symbols.

   Actually, Scheme (which admittedly isn't a real Lisp by many
standards) doesn't have get/put at all.  And good riddance, I might
add:-)

   From: "Stephen J. Turnbull" <stephen@xemacs.org> Date: 07 May 2002
20:04:50 +0900

   >>>>> "ms" == Michael Sperber <sperber@informatik.uni-tuebingen.de>
writes:

   Stephen> special functions (coding-system-get,
coding-system-property, Stephen> charset-property are examples I find
particularly obnoxious).  I would Stephen> like to make these obsolete
by allowing `get' on charsets, coding Stephen> systems, etc.

   ms> But I don't see why it has to be GET that accesses the plist.
 ms> You just build more dispatch into GET with no immediate     ms>
benefit to the API.  Ad-hoc genericity gets you something when     ms>
there's some place in the code you don't know what the     ms>
underlying object is.  I don't see this being the case here.      ms>
Why do you find them "obnoxious?"

   Their semantics are basically `get'.  Why not use that name?  Of
course I agree that it doesn't have to be `get', but why clutter things
up?

   But those are particularly obnoxious because of the object/name
confusion they have built in.  Ie, my real problem with them is more
ancient Mule idiom than the *-get or *-property names for the API.

   ms> Actually, Scheme (which admittedly isn't a real Lisp by many
ms> standards) doesn't have get/put at all.

   What does it use instead?  (And no, you can't bait _me_ with Lisp
definition trolls, I think of XML as "declarative LISP with fat,
flavored, fuzzy parentheses.")

   From: sperber@informatik.uni-tuebingen.de (Michael Sperber [Mr.
Preprocessor]) Date: Tue, 07 May 2002 13:26:13 +0200

   Stephen> Their semantics are basically `get'.  Why not use that name?

   Because it doesn't convey as much information in the source code as
it could, and because it provides less type checking than it could.

   Stephen> What does it use instead?

   What for?  I've never felt the desire to use them, and it seems to me
that in Lisp, properties are usually used for one of two purposes:

   - As a poor man's replacement for hash tables.

   - To store data which should really be stored inside the object
itself.

   In the former case, I use a hash table.  In the latter case, I store
the data in the object itself.


File: internals.info,  Node: Discussion -- Switching to C++,  Next: Discussion -- Windows External Widget,  Prev: Discussion -- Instantiators and Generic Property Accessors,  Up: Future Work Discussion

45.6 Discussion - Switching to C++
==================================

From: "Ben Wing" <ben@666.com> Date: Fri, 10 May 2002 19:42:53 -0700

   i know i'm opening up a bag of worms by suggesting this, but what
about moving to C++?  I know others advocate this (Jan, Martin), and
the more I read Stroustrup's 3rd edition, the more I realize that
*HUGE* armounts of code in XEmacs, and in particular most of the really
hairy and hard-to-understand stuff - lots of weird macros, faux
object-oriented stuff implemented in multiple places, each differently
(Lisp objects; methods on consoles/devices/etc; specifier sub-types;
coding-system sub-types; image-instance device methods; image-instance
format methds; etc.), all the GCPROS (which could go entirely),
dynarrs, eistring, etc. etc. - is simply superseded by stuff already
built into C++ or supplied by the standard libraries.  Just now, I was
going through the redisplay code, and noticing the huge amount of
duplication between gtk and X, something that's hard to fix [except
through super ad-hoc ways like using a .c file as a .h file to
"generate" lots of similar but slightly different code] in C, but is
extremely easy in C++ using inheritance [and/or templates].  for
example, instead of having just one layer of device methods, you'd have

     general -> tty
                -> windowing -> mswindows
                                    -> xlike -> x, gtk

   which would nicely and naturally encapsulate lots of duplicated [and
thus, hard to maintain] code.

   even more of a win would be the GCPRO's.  Taking advantage of
constructors and destructors, we could simply do away [COMPLETELY!]
with explicitly gcproing, and still have everything gcpro'd. [in fact,
much more reliably - none of the dreaded "temporary" problem, and every
reference is always gcpro'd so we have greater flexibility for GC work
- take note, Michael :-) - e.g. we could safely garbage collect when
allocating, and we could even implement a relocating garbage collector.
in the few places where performance might be an issue [i seriously
doubt there'd be many of them], we simply use a separate
Lisp_Object_No_GCPRO class (presumably a base class of Lisp_Object),
and manually handle the GCPRO's ourselves.  If we needed to distinguish
here between static and dynamic objects, or static vs. local vs. heap,
we could do so easily with bit flags in the object pointed to - we have
space for lots of them.

   code reliability and maintainability would likely substantially
increase due to the ability to express most things in a natural C++ way
instead of lots of weird hackish hard-to-understand C stuff
implementing stuff the language wasn't really designed for.
Furthermore, there are even some possibilities for increased speed -
many operations that can only reasonably be done now using Lisp objects
(and the associated gc overhead and such) could be done using the
high-level built-in facilities of C++, which in their ease of use
approach Lisp; and C++ has `inline' built-in, so we could easily add
various container classes to improve the understandability of the code
without loss of performance.

   finally, making the "switch" is trivial, since martin did the initial
work making XEmacs C++-safe and I've been keeping it that way - I
regularly build under C++ and fix any problems.  All we'd need to do is
switch the compiler and start gradually introducing C++ constructs as
we feel like it.

   for those concerned that dumping might stop working, [a] i don't
think it would, [b] the portable dumper has come of age - i use it
almost all the time, and it's rock-solid and not obviously slower than
unexec.

   the only major concern that i see is the quality of the C++
implementations out there, in particular G++, which is the most widely
available.  I know that 6 years ago G++ was a bit rocky - I went to
interview for Netscape, and they mentioned having to rely on various
vendor implementations of C++, whereas they would have preferred G++ if
it was reliable, due to the sameness of environment.  But that was *SIX
YEARS* ago!  Stroustrup 3rd Edition has been out for 5 years now, and
it defines, as far as I know, ANSI Standard C++ - so that's at least 5
years to implement a standard.  It's hard to believe that G++ isn't
completely reliable now; but I do not have as much experience as others.

   What do you think?  I would *really* like to make this change, as it
would immensely facilitate lots of code I'm working on and will be
working on, plus of course add all the above benefits once we get
around to converting the code.

   From: "Stephen J. Turnbull" <stephen@xemacs.org> Date: 11 May 2002
15:34:08 +0900

   I don't have a real problem with it, as long as we're very
conservative about it, ie, using C++ as "clean C with classes", and
introducing things slowly.  Implement everything ourselves, avoid the
standard class libraries.

   I've been following the Python lists recently, and although the bias
is easy to guess, it's interesting to note that the people who are most
anti-C++ are typically the ones who are world-class C++ programmers
with big projects under their belts.  Many of them actually advocate
using C rather than C++.

   I do worry that with Martin currently out of the picture we don't
have an active C++ standards bigot and implementation collector to deal
with compiler-specific issues.  We do OK with C, but C++ is a much more
complex, subtle language.  Is there anybody else to plausibly take on
that role?

   From: Hrvoje Niksic <hniksic@arsdigita.com> Date: Sun, 12 May 2002
20:58:50 +0200

   I'm strongly opposed to this.  Here are some reasons:

   * C++ may fix some problems, but it will introduce others, some of
which may be much harder to fix.  XEmacs is already a large program,
hard to understand.  C++ will not improve things.

   * XEmacs will suddenly become uncompilable and unusable in   many
environments where it used to build perfectly fine - for   example,
those that don't ship with a C++ compiler at all.  We could   "make GCC
3 a requirement", but I don't like that idea.

   * People without C++ experience will no longer be able to hack
XEmacs.    I'd be the first one to leave.  For example, I know quite a
few   programmers who don't care for Qt and KDE simply because it's C++.

   * C is the /lingua franca/ of free software development.  If we're
switching languages, it should be for a good reason and to something
we agree is an improvement (e.g. Common Lisp).

   * C++ is not the be-all end-all to everything.  People who undestand
 it well are usually the first ones to warn against it.  It's
possible that they were scarred by using C++ at a bad time, but I'd
think twice before discounting their advice or blindly believing   that
C++ is now all better.

   If you were writing a new project, I'd say go for it.  But at this
point, this seems like a needless tweak.  Do we really need *more*
internal reorganizations?  Shouldn't we work on user-visible features?
Wasn't that what you yourself advocated when I talked to you?

   From: Didier Verna <didier@xemacs.org> Date: Tue, 14 May 2002
11:21:32 +0200

   Switching to C++ has been suggested for the first time at the M17n'99
conference in Japan IIRC. Although it was around a table with many empty
bottles of beer on it :-), I've kept some hope from that time. I'm
happy to see Ben in favor of this today. This idea coming from him is
likely to have more impact than when it comes from me or Yan of whoever
else.

   There are several points that make me in favor of this change:

   - C++ support is already there thanks to Martin.

   - The amount of OO simulation code written in C in XEmacs is *HUGE*.
But more   important, this code simulates *BASIC* OO features that are
not any more a   problem for any C++ compiler. I mean, by just using
the basic features of   C++ in terms of OO and data abstraction
(classes, inheritance, methods (with   inlining), operators
overloading), we'll win big time in code size,   readability,
maintainability, and correctness.

   - the fact that *basic* OO support is already a major gain is very
important   to me. You don't have to go generic programming with
templates everywhere to   write an OO XEmacs[1]. Switching to C++ can
be completely gradual, and we   can even stop early in the C++ features
we use. That will already be a big   gain. That's also the advantage
over the idea of using another more modern   language to rewrite the
core; something completely unrealistic.

   - there is another important aspect on the design issue: many people
 (including from the industry) have worked on abstracting common
problems in   an OO philosophy. Some people claim that the concepts
that emerged from this   kind of work of just C++ specific hackery, and
they're probably right, but   anyway that's obviously not a problem for
us. Any C++ writer should have the   "Design Patterns" book in hand. It
already has good design solutions for   many problems that we're facing
in XEmacs (like, supporting more than one   widget set), because these
problems are so *common*. By using C++ we can   directly benefit from
the experience of other large applications designers.

   Footnotes: [1]  We're working on GP in C++ in our lab here and we
trigger bugs in gcc 3.       But you should see the code in question,
it's pure template and static      programming. Things that XEmacs will
never need.

   From: Daniel Pittman <daniel@rimspace.net> Date: Sat, 11 May 2002
19:15:04 +1000

   I've been following the Python lists recently, and although the bias
> is easy to guess, it's interesting to note that the people who are >
most anti-C++ are typically the ones who are world-class C++ >
programmers with big projects under their belts.  Many of them >
actually advocate using C rather than C++.

   I wouldn't class myself as "world-class", but I can understand this
perspective based on my experiences with large projects that aim for
portability to vendor compilers, not just gcc.

   The biggest problem, assuming that you are willing to ignore
platforms like Sinix-PC[1] and their poor compiler support[2] is that
it's easy to shoot yourself in the foot with C++.

   The biggest portability problems are namespaces, the standard C++
library and template support, in about that order, followed by exception
handling.

   Very few things get namespaces right, even today, with gcc being one
of the worst. Tempting as they are, they are best avoided where
possible, except in compiler and platform specific code.[3]

   The standard C++ library, which supports RTTI and a few other things
including the [io]stream tools, is less than totally reliable although
it can be used with relative safety most places.

   What you really need to watch out for with that is the extensions
that every vendor in the universe has added to their collections because
there isn't any standard way of doing common things in most of these
areas.

   The Standard Template Library isn't. Aside from a tendency to expose
limitations of symbol name lengths[4] the library tends to be unreliable
in behavior between compilers and platforms. Not enough to make simple
things fail, though, just enough to make it occasionally do odd things
or show up obscure bugs in your code...

   It's also not very well designed, I think, as libraries go. That's a
personal opinion, though, and not universal.

   C++ exceptions are an interesting issue. They can work extremely
well as a mechanism for managing errors and improve the reliability of
the system.

   They can also become an unending nightmare of epic proportions,
causing more pain and suffering than you can imagine. :/

   The main difference between the two situations, so far as I can tell,
comes from two aspects of design that have ... far reaching
implications.

   If you try adding exceptions to code that isn't ready to deal with
them, things tend to go very badly wrong. I /think/ that the existing
exception mechanisms in XEmacs would be similar enough that this isn't
the case, though.

   The other is that you need to base your code very strongly around the
"construction acquires, destruction releases" model of resource
handling. This, of course, implies using exceptions everywhere because
you /can't/ use that model in C++ without them.[5]

   Again, I think that the existing XEmacs model will probably work well
with this, but I am hardly an expert at either; my only real-world
experience is the one project where I gained these impressions and the
knowledge of the suffering they can bring. :)

   Oh, and finally, watch out for operator overloading - including
casts.  These are very easy to abuse into a position where your code is
impossible for others to understand.

   I would also advocate avoiding multiple inheritance, but that's
because my personal design experience says that it's almost always a
sign of bad design. Views there vary greatly.

   > I do worry that with Martin currently out of the picture we don't
have > an active C++ standards bigot and implementation collector to
deal > with compiler-specific issues.

   You probably have more need of the second than the first. There are
not many things you actually need a standards bigot for; just write
good C and don't use too many things other than classes.

   > We do OK with C, but C++ is a much more complex, subtle language.

   C with classes, or the limited subset of C++ that doesn't include
templates, exceptions or RTTI is not much more complex than standard C.

   If you add exceptions to that you will probably not notice anything
but a syntax change in the core, given their current standing. Er, they
probably don't work right in signal handlers, though, because they don't
know anything about them.[6]

   > Is there anybody else to plausibly take on that role?

   I would be happy to look at things that were publicly discussed on
the topic but I don't think I have the experience or the knowledge of
the XEmacs development process to do anything more than that.

   Not, I imagine, that anyone would ask. :)

   Daniel

   Footnotes: [1]  Archaic Unix ported to i386 from a minicomputer over
a decade ago.

   [2]  The vendor C++ would segfault on anything that had multiple
inheritance. :)

   [3]  I found them invaluable in resolving a few Win9x vn WinNT symbol
    conflicts, for example, but that's obviously target-specific.

   [4]  The current record for STL-generated name length that I have
seen      is a symbol 892 characters long...

   [5]  The lack of a return code from a class constructor is the killer
    issue here.

   [6]  This, I believe, varies from vendor to vendor. :)


File: internals.info,  Node: Discussion -- Windows External Widget,  Next: Discussion -- Packages,  Prev: Discussion -- Switching to C++,  Up: Future Work Discussion

45.7 Discussion - Windows External Widget
=========================================


     Subject:
                 Re: External Widget Support for Xemacs on nt
            Date:
                 Sat, 08 Jul 2000 01:47:14 -0700
           From:
                 Ben Wing <ben@666.com>
             To:
                 Timothy.Fowler@msdw.com
             CC:
                 xemacs-nt@xemacs.org
      References:
                 1




     Nothing is currently done for external widget support under XEmacs but
     it should not be too hard to do and would be a great addition to
     XEmacs. What you would probably want to do is create an XEmacs control
     that has an interface something like the built-in edit control and
     which communicates to an existing XEmacs process using DDE. (Basically
     you would modify XEmacs so that it registered itself as a DDE server
     accepting external widget requests, and then the external edit control
     would simply send a DDE request and the result would be a handle of
     some sort used for future communication with that particular XEmacs
     process.)

     There are two basic issues in getting the external widget to work,
     which are display and input. Although I am not completely sure, I have
     a feeling that it is possible for one process to write into the window
     of another process, simply by using that window's HWND handle. If so
     it should be extremely easy to get the output working (this is exactly
     the approach used under Xt). For input, you would probably again want
     to do what is done under Xt, which is that the client widget simply
     passes all of the appropriate messages to the XEmacs server process
     using whatever communication channel was set up, e.g. DDE, and the
     XEmacs server processes them normally. Very few modifications would be
     needed to the XEmacs source code and all of the necessary
     modifications could be done simply by looking for existing external
     widget code in XEmacs.

     If you are interested in continuing this, I will certainly give you
     any support you need along the way. This would be a great project to
     be added to XEmacs.



     Timothy Fowler wrote:

     > I am looking into external widget support for xemacs nt similar to that
     > existing in xemacs for X
     > Have any developement efforts been made in this direction in the past?
     > Is there any current effort?
     > Any insight into the complexity of achieving this?
     > Any comments would be greatly appreciated
     > Thanks
     > Tim Fowler

     --
     Ben

     In order to save my hands, I am cutting back on my mail.  I also write
     as succinctly as possible -- please don't be offended.  If you send me
     mail, you _will_ get a response, but please be patient, especially for
     XEmacs-related mail.  If you need an immediate response and it is not
     apparent in your message, please say so.  Thanks for your understanding.

     See also http://www.666.com/ben/chronic-pain/


     Subject:
             RE: External Widget Support for Xemacs on nt
        Date:
             Mon, 10 Jul 2000 12:40:01 +0100
        From:
             "Alastair J. Houghton" <ajhoughton@lineone.net>
          To:
             "Ben Wing" <ben@666.com>, <xemacs-nt@xemacs.org>
         CC:
             <Timothy.Fowler@msdw.com>




     > -----Original Message-----
     > From: owner-xemacs-nt@xemacs.org [mailto:owner-xemacs-nt@xemacs.org]On
     > Behalf Of Ben Wing
     > Sent: 08 July 2000 09:47
     > To: Timothy.Fowler@msdw.com
     > Cc: xemacs-nt@xemacs.org
     > Subject: Re: External Widget Support for Xemacs on nt
     >
     > Nothing is currently done for external widget support under
     > XEmacs but it should
     > not be too hard to do and would be a great addition to XEmacs.
     > What you would
     > probably want to do is create an XEmacs control that has an
     > interface something
     > like the built-in edit control and which communicates to an
     > existing XEmacs
     > process using DDE.

     It would be *much* better to use RPC or COM rather than DDE - and
     also it would provide a more useful interface to XEmacs (like the
     Microsoft rich text edit control that is used by Wordpad). It
     would probably also be easier...

     > If you are interested in continuing this, I will certainly give
     > you any support
     > you need along the way. This would be a great project to be added
     > to XEmacs.

     I agree. This would be a *really useful* thing to do...

     Regards,

     Alastair.

     ____________________________________________________________
     Alastair Houghton                     ajhoughton@lineone.net

     Subject:
                 Re: External Widget Support for Xemacs on nt
            Date:
                 Mon, 10 Jul 2000 22:56:06 -0700
           From:
                 Ben Wing <ben@666.com>
             To:
                 "Alastair J. Houghton" <ajhoughton@lineone.net>
             CC:
                 xemacs-nt@xemacs.org, Timothy.Fowler@msdw.com
      References:
                 1




     sounds good.  i don't know too much about windows ipc methods, so i suggested
     dde just as an example.

     "Alastair J. Houghton" wrote:

     > > -----Original Message-----
     > > From: owner-xemacs-nt@xemacs.org [mailto:owner-xemacs-nt@xemacs.org]On
     > > Behalf Of Ben Wing
     > > Sent: 08 July 2000 09:47
     > > To: Timothy.Fowler@msdw.com
     > > Cc: xemacs-nt@xemacs.org
     > > Subject: Re: External Widget Support for Xemacs on nt
     > >
     > > Nothing is currently done for external widget support under
     > > XEmacs but it should
     > > not be too hard to do and would be a great addition to XEmacs.
     > > What you would
     > > probably want to do is create an XEmacs control that has an
     > > interface something
     > > like the built-in edit control and which communicates to an
     > > existing XEmacs
     > > process using DDE.
     >
     > It would be *much* better to use RPC or COM rather than DDE - and
     > also it would provide a more useful interface to XEmacs (like the
     > Microsoft rich text edit control that is used by Wordpad). It
     > would probably also be easier...
     >
     > > If you are interested in continuing this, I will certainly give
     > > you any support
     > > you need along the way. This would be a great project to be added
     > > to XEmacs.
     >
     > I agree. This would be a *really useful* thing to do...
     >
     > Regards,
     >
     > Alastair.
     >
     > ____________________________________________________________
     > Alastair Houghton                     ajhoughton@lineone.net

     --
     Ben

     In order to save my hands, I am cutting back on my mail.  I also write
     as succinctly as possible -- please don't be offended.  If you send me
     mail, you _will_ get a response, but please be patient, especially for
     XEmacs-related mail.  If you need an immediate response and it is not
     apparent in your message, please say so.  Thanks for your understanding.

     See also http://www.666.com/ben/chronic-pain/


File: internals.info,  Node: Discussion -- Packages,  Next: Discussion -- Distribution Layout,  Prev: Discussion -- Windows External Widget,  Up: Future Work Discussion

45.8 Discussion - Packages
==========================

Author: Ben Wing (mailto:ben@xemacs.org)

Important package-related changes
---------------------------------

This file details changes that make the package system no longer an
unmitigated disaster.  This way, at the very least, people can
essentially ignore the package system and not get bitten horribly the
way they currently do.

  1. A single tarball containing absolutely everything and named
     xemacs-21.2.68.tar.gz.  This must contain absolutely everything,
     including all of the packages, and in the proper directory
     structure, so that the paradigm for

     untar; configure; make; make install

     just works.

  2. Fixed startup slowdown when all packages are installed so that
     there is absolutely no penalty to having them all installed.  This
     may be hard.

  3. All files on the ftp site should be accessible through http.

  4. Put symlinks into the distribution directory to the appropriate
     files in the package directory.

  5. Eliminate the confusing SUMO name, choosing a much more obvious
     name such as all-packages.

  6. There should be no separation of mule and non-mule packages.

  7. Having 2 packages that conflict with each other should be
     completely disallowed.

  8. Fix vc and ps-print so that there is only ONE version.

  9. Fix up all of the READMEs on the distribution site to make it
     abundantly clear what needs to be obtained, where to get it, and
     how to install it, especially with regards to packages.


File: internals.info,  Node: Discussion -- Distribution Layout,  Prev: Discussion -- Packages,  Up: Future Work Discussion

45.9 Discussion - Distribution Layout
=====================================

     From:
             Ben Wing <ben@666.com>
                                                          10/15/1999 8:50 PM

      Subject:
             VOTE: Absolutely necessary changes to file naming in releases
          To:
             SL Baur <steve@xemacs.org>,
             XEmacs Reviews <xemacs-review@xemacs.org>




     Everybody except Steve seems to agree that we need to provide a single
     tar file containing the entire XEmacs tree whenever we release a new
     version of XEmacs (beta or not).  Therefore I propose the following
     simple changes, and ask for a vote.  If it is the general will of the
     developers, then Steve *WILL* make these changes.  This is the
     definition of cooperative development -- no one, not even the
     maintainer, can assert absolute power over anything.

     I propose (assuming, for example, release 21.2.20):

     1. xemacs-21.2.20.tar.gz -> xemacs-21.2.20-core.tar.gz

     2. xemacs-sumo.tar.gz -> xemacs-packages.tar.gz

     3. xemacs-mule-sumo.tar.gz -> xemacs-mule-packages.tar.gz

     4. Symlinks to the files mentioned in #2 and #3 get created in the SAME
     directory as xemacs-21.2.20-*.tar.gz.

     5. MOST IMPORTANTLY, a new file xemacs-21.2.20.tar.gz gets created,
     which is the combination of the 5 files xemacs-21.2.20-core.tar.gz,
     xemacs-21.2.20-elc.tar.gz, xemacs-21.2.20-info.tar.gz,
     xemacs-packages.tar.gz, and xemacs-mule-packages.tar.gz.


     The directory structure of the new combined file xemacs-21.2.20.tar.gz
     would look like this:

     xemacs-21.2.20/
     xemacs-packages/
     xemacs-mule-packages/


     I am sorry to shout, but the current situation is just completely
     insane.

     ben






     From:
             Ben Wing <ben@666.com>
                                                         10/16/1999 3:12 AM

      Subject:
             Re: VOTE: Absolutely necessary changes to file naming in releases
          To:
             SL Baur <steve@xemacs.org>,
             XEmacs Reviews <xemacs-review@xemacs.org>,
             "Michael Sperber [Mr. Preprocessor]" <sperber@informatik.uni-tuebingen.de>




     Something went wrong with my mail program while I was responding, so
     Michael's response is not quoted here.

     Let me rephrase my proposal, stressing the important points in order of
     importance:

     1. MOST IMPORTANT: There MUST be a SINGLE tar file containing the complete
     XEmacs sources, packages, etc.  The name of this tar file must have a
     format like this:

     xemacs-21.2.10.tar.gz

     The directory layout of the packages within it is not important as long as
     it works: The user who downloads the tar file MUST be able to apply the
     'configure; make; make install' paradigm at the top-level directory and
     have it work properly.

     2. All the pieces of XEmacs must be in the *same* subdirectory on the FTP
     site.

     3. The names need to be obvious and standard.  Naming the core files
     "xemacs-21.2.20.tar.gz" is non-standard because those are only the core
     files.  The standard followed by everybody in the world is that a name like
     this refers to the entire product, with all ancillary files.  Also, "sumo",
     although a nice in-joke, is extremely confusing and needs to go.

     Referring to Michael's point about the layout I proposed, I also think that
     the package system needs to be modified to accept a layout produced by the
     "obvious" way of obtaining and untarring the parts, which leaves you with a
     directory consisting of

     xemacs-21.2.19/
     xemacs-packages/
     mule-packages/

     All at the same level.  However, this is an independent issue from the vote
     at hand.


     Consider the current insanity.  The new XEmacs user or beta tester goes to
     the FTP site, looks around, finds the file xemacs-21.2.19.tar.gz, and
     downloads it, because it looks like the obvious one to get.  But it doesn't
     work.  Oops ...  He looks some more and finds the other two -elc and -info
     parts, grabs them, and then tries again.  But it still doesn't work.  He
     manages to overhear something about packages, so he looks for them, but
     doesn't find them immediately (they're not even in the beta tree, though
     they obviously contain beta-level code, especially in xemacs-base and
     mule-base).  Eventually he discovers the package/ subdirectory, but what
     the hell does he do there?  There's no README at all there giving any
     clues, so he downloads everything.  Along with this, he gets some files
     called "sumo", which he doesn't understand, but he notices that some of
     them are extremely large.  "sumo" ... "large" ...  hehe, I get it.  Some
     silly developer's joke.  But then he tries again to compile things, and
     just can't figure things out.  He still doesn't know:

     -- "sumo" is not just some large file, but is a tar file of all the
     packages.
     -- The packages can't be placed is any subdirectory in any obvious relation
     to the XEmacs directory ("straight out of the box" if you manage to grok
     the significance of the sumo files, you get a layout like

     xemacs-21.2.19/
     xemacs-packages/
     mule-packages/

     which naturally doesn't work!  He needs to put them underneath
     xemacs-21.2.19/lib/xemacs/ or something.)

     At this point, he gives up, and (if he was a user of a pre-packagized
     XEmacs) wonders in despair how things got so messed up, when all older
     XEmacs releases, including all the betas, followed the standard "configure;
     make; make install" paradigm).



     Soooooo .........  PLEASE vote on issues #1-3 above, and add any comments
     you feel like adding.

     ben

     Ben Wing wrote:

     > Everybody except Steve seems to agree that we need to provide a single
     > tar file containing the entire XEmacs tree whenever we release a new
     > version of XEmacs (beta or not).  Therefore I propose the following
     > simple changes, and ask for a vote.  If it is the general will of the
     > developers, then Steve *WILL* make these changes.  This is the
     > definition of cooperative development -- no one, not even the
     > maintainer, can assert absolute power over anything.
     >
     > I propose (assuming, for example, release 21.2.20):
     >
     > 1. xemacs-21.2.20.tar.gz -> xemacs-21.2.20-core.tar.gz
     >
     > 2. xemacs-sumo.tar.gz -> xemacs-packages.tar.gz
     >
     > 3. xemacs-mule-sumo.tar.gz -> xemacs-mule-packages.tar.gz
     >
     > 4. Symlinks to the files mentioned in #2 and #3 get created in the SAME
     > directory as xemacs-21.2.20-*.tar.gz.
     >
     > 5. MOST IMPORTANTLY, a new file xemacs-21.2.20.tar.gz gets created,
     > which is the combination of the 5 files xemacs-21.2.20-core.tar.gz,
     > xemacs-21.2.20-elc.tar.gz, xemacs-21.2.20-info.tar.gz,
     > xemacs-packages.tar.gz, and xemacs-mule-packages.tar.gz.
     >
     > The directory structure of the new combined file xemacs-21.2.20.tar.gz
     > would look like this:
     >
     > xemacs-21.2.20/
     > xemacs-packages/
     > xemacs-mule-packages/
     >
     > I am sorry to shout, but the current situation is just completely
     > insane.
     >
     > ben



     From:
             Ben Wing <ben@666.com>
                                                          12/6/1999 4:19 AM

      Subject:
             Re: Please Vote on Proposals
          To:
             Kyle Jones <kyle_jones@wonderworks.com>
         CC:
             XEmacs Review <xemacs-review@xemacs.org>




     OK Kyle, how about a different proposal:

     1. The distribution consists of the following three parts (let's assume
     v21.2.25):

     -- xemacs-21.2.25-core.tar.gz
        The same as would currently in xemacs-21.2.25.tar.gz.  You can
        run this editor and edit in fundamental mode, but not do anything
     else.

     -- xemacs-21.2.25-core-packages.tar.gz
        A useful and complete subset of all the possible packages.  Selection
     of
        what goes in and what goes out is based partially on consensus,
     partially
        on vote, and partially on these criteria:

         -- commonly-used packages go in.
         -- unmaintained or out-of-date packages go out.
         -- buggy, poorly-written packages go out.
         -- really obscure packages that hardly anybody could possibly care
            about go out.
         -- when there are two or three packages implementing basically the
            same functionality, pick only one to go in unless there are two
     that
            both are really commonly-used.
         -- if a package can be loaded implicitly as a result of something in
     the
            core, it needs to go in, regardless of whether it's been
     maintained.
            This applies, for example, to the mode files -- *all* mode
     packages must
            go in (or more properly, every mode must have a corresponding
     package
            that's in, although if there are two or more packages implementing
     a
            particular mode, e.g. html, we are free to choose just one).

     -- xemacs-21.2.25-aux-packages.tar.gz
        All of the packages not in the previous file.  Generally
     crappy-quality,
        poorly-maintained code.

     Note, we do not make distinctions between Mule and non-Mule in our
     packaging scheme -- this is a bug and XEmacs and/or the packages should
     be fixed up so that this goes away.

     2. The distribution also contains two combination files:

     -- xemacs-21.2.25.tar.gz
        This is the "default" file that a naive user ought to retrieve, and
        he'll get a running XEmacs, just like he wants, and comfortable, too,
        because all of the common packages are there.  This file is a
     combination
        of xemacs-21.2.25-core.tar.gz and xemacs-21.2.25-core-packages.tar.gz.

     -- xemacs-21.2.25-everything.tar.gz
        This file contains absolutely everything, like it advertises --
        including the aux packages and all of their associated crappy-quality,

        unmaintained code.  This file is a combination of
     xemacs-21.2.25-core.tar.gz,
        xemacs-21.2.25-core-packages.tar.gz, and
     xemacs-21.2.25-aux-packages.tar.gz.


     I like this proposal better than the previous one I advocated, because it
     follows your good suggestion of separating the wheat from the chaff in
     the packages, so to speak.  People will grab xemacs-21.2.25.tar.gz by
     default, just like they should,
     and they'll get something they're quite happy with, and we're happy
     because we can exercise quality control over the packages and exclude the
     crappy ones most likely to cause grief later on.


     What say y'all?

     ben



     Kyle Jones wrote:

     > Ben Wing writes:
     >  > Disagree.  Please let's follow everyone else's convention, and not
     >  > introduce yet another randomness.
     >
     > It is not randomness! I think this is a semantic issue and an
     > important one.  The issue is: What do we consider part of XEmacs
     > and what is considered external to XEmacs.  If you put all the
     > packages in xemacs.tar.gz, then users can reasonably and wrongly
     > assume that all this random Lisp code is maintained by us.  We
     > are trying to stay away from that model because in the past it has
     > left us with piles and piles of orphaned code.  Even if every one
     > of us were paid to maintain XEmacs, it is just not practical for
     > us to continue to maintain all that code, let alone any new code.
     > So I think the naming distinction Jan is making is worth doing.
     >
     > Also, I don't consider the current situation broken, except
     > perhaps the sumo tarball being out of date.  I never, ever,
     > though it was a great idea to ship all the stuff that XEacs
     > shipped in the old days.  Because this pile of code was always
     > around in the distribution, an enormous web of undocumented
     > dependencies was constructed.  Eventually, you HAD to install
     > everything because if you left something out or removed something
     > you never knew when XEmacs would throw an error.  Thus the Cult
     > of the Cargo was born.
     >
     > One of the best things that came out of the package system was
     > the month or two we spent running XEmacs without all the assorted
     > Lisp installed.  Dependencies were removed or documented, some
     > stuff got retired, and for the first time we actually had a full
     > accounting of what we were shipping.  I currently run XEmacs with
     > 7 packages and I don't miss the other stuff.
     >
     > Having come this far, I do not think we should go back to
     > advocating that everyone just install everything and not
     > think about they are doing.  Besides saving space and startup
     > time, another reason to not install everything is that you
     > won't bloat your XEmacs process nearly as much if you go
     > exploring in the Custom menus, because there won't be as much
     > Lisp loaded as Custom sets up its groups and whatnot.

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it is not apparent in your message,
     please say so.  Thanks for your understanding.


File: internals.info,  Node: Old Future Work,  Next: Index,  Prev: Future Work Discussion,  Up: Top

46 Old Future Work
******************

This chapter includes proposals for future work that were later
implemented.  These proposals are included because they may describe to
some extent the actual workings of the implemented code, and because
they may discuss relevant design issues, alternative implementations, or
work still to be done.

* Menu:

* Old Future Work -- A Portable Unexec Replacement::
* Old Future Work -- Indirect Buffers::
* Old Future Work -- Improvements in support for non-ASCII (European) keysyms under X::
* Old Future Work -- RTF Clipboard Support::
* Old Future Work -- xemacs.org Mailing Address Changes::
* Old Future Work -- Lisp callbacks from critical areas of the C code::
* Old Future Work -- Everything should obey duplicable extents::
* Old Future Work -- Display Tables::


File: internals.info,  Node: Old Future Work -- A Portable Unexec Replacement,  Next: Old Future Work -- Indirect Buffers,  Prev: Old Future Work,  Up: Old Future Work

46.1 Old Future Work - A Portable Unexec Replacement
====================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* Currently, during the build stage of XEmacs, a bare
version of the program (called "temacs") is run, which loads up a bunch
of Lisp data and then writes out a modified executable file.  This
process is very tricky to implement and highly system-dependent.  It can
be replaced by a simple, mostly portable, and easy to implement scheme
where the Lisp data is written out to a separate data file.

   The scheme makes only three assumptions about the memory layout of a
running XEmacs process, which, as far as I know, are met by all current
implementations of XEmacs (and they're also requirements of the existing
unexec scheme):

  1.  The initialized data segments of the various XEmacs modules are
     all laid out contiguously in memory and are separated from the
     initialized data segments of libraries that are linked with
     XEmacs; likewise for uninitialized data segments.

  2.  The beginning and end of the XEmacs portion of the combined
     initialized data segment can be programmatically determined;
     likewise for the uninitialized data segment.

  3.  The XEmacs portion of the initialized and uninitialized data
     segments are always loaded at the same place in memory.


   Assumption number three means that this scheme is non-relocatable,
which is a disadvantage as compared to other, relocatable schemes that
have been proposed.  However, the advantage of this scheme over them is
that it is much easier to implement and requires minimal changes to the
XEmacs code base.

   First, let's go over the theory behind the dumping mechanism.  The
principles that we would like to follow are:

  1.  We write out to disk all of the data structures and all of their
     sub-structures that we have created ourselves, except for data
     that is expected to change from invocation to invocation (in
     particular, data that is extracted from the external environment
     at run time).

  2.  We don't write out to disk any data structures created or
     initialized by system libraries, by the kernel or by any other
     code that we didn't create ourselves, because we can't count on
     that code working in the way that we want it to.

  3.  At the beginning of the next invocation of our program, we read
     in all those data structures that we have written out to disk, and
     then continue as if we had just created and initialized all of
     that data ourselves.

  4.  We make sure that our own data structures don't have any pointers
     to system data, or if they do, that we note all of these pointers
     so that we can re-create the system data and set up pointers to
     the data again in the next invocation.

  5.  During the next invocation of our program, we re-create all of
     our own data structures that are derived from the external
     environment.


   XEmacs, of course, is already set up to adhere to most of these
principles.

   In fact, the current dumping process that we are replacing does a
few of these principles slightly differently and adds a few extra of
its own:

  1.  All data structures of all sorts, including system data, are
     written out.  This is the cause of no end of problems, and it is
     avoidable, because we can ensure that our own data and the system
     data are physically separated in memory.

  2.  Our own data structures that we derive from the external
     environment are in fact written out and read in, but then are
     simply overwritten during the next invocation with new data.
     Before dumping, we make sure to free any such data structure that
     would cause memory leaks.

  3.  XEmacs carefully arranges things so that all static variables in
     the initialized data are never written to after the dumping stage
     has completed.  This allows for an additional optimization in
     which we can make static initialized data segments in pre-dumped
     invocations of XEmacs be read-only and shared among all XEmacs
     processes on a single machine.


   The difficult part in this process is figuring out where our data
structures lie in memory so that we can correctly write them out and
read them back in.  The trick that we use to make this problem solvable
is to ensure that the heap that is used for all dynamically allocated
data structures that are created during the dumping process is located
inside the memory of a large, statically declared array.  This ensures
that all of our own data structures are contained (at least at the time
that we dump out our data) inside the static initialized and
uninitialized data segments, which are physically separated in memory
from any data treated by system libraries and whose starting and ending
points are known and unchanging (we know that all of these things are
true because we require them to be so, as preconditions of being able to
make use of this method of dumping).

   In order to implement this method of heap allocation, we change the
memory allocation function that we use for our own data.  (It's
extremely important that this function not be used to allocate system
data.  This means that we must not redefine the `malloc' function using
the linker, but instead we need to achieve this using the C
preprocessor, or by simply using a different name, such as `xmalloc'.
It's also very important that we use the correct `free' function when
freeing dynamically-allocated data, depending on whether this data was
allocated by us or by the


File: internals.info,  Node: Old Future Work -- Indirect Buffers,  Next: Old Future Work -- Improvements in support for non-ASCII (European) keysyms under X,  Prev: Old Future Work -- A Portable Unexec Replacement,  Up: Old Future Work

46.2 Old Future Work - Indirect Buffers
=======================================

Author: Ben Wing (mailto:ben@xemacs.org)

   An indirect buffer is a buffer that shares its text with some other
buffer, but has its own version of all of the buffer properties,
including markers, extents, buffer local variables, etc.  Indirect
buffers are not currently implemented in XEmacs, but they are in GNU
Emacs, and some people have asked for this feature.  I consider this
feature somewhat extent-related because much of the work required to
implement this feature involves tracking extents properly.

   In a world with indirect buffers, some buffers are direct, and some
buffers are indirect.  This only matters when there is more than one
buffer sharing the same text.  In such a case, one of the buffers can be
considered the canonical buffer for the text in question.  This buffer
is a direct buffer, and all buffers sharing the text are indirect
buffers.  These two kinds of buffers are created differently.  One of
them is created simply using the `make_buffer()' function (or perhaps
the `Fget_buffer_create()' function), and the other kind is created
using the `make_indirect_buffer()' function, which takes another buffer
as an argument which specifies the text of the indirect buffer being
created.  Every indirect buffer keeps track of the direct buffer that
is its parent, and every direct buffer keeps a list of all of its
indirect buffer children.  This list is modified as buffers are created
and deleted.  Because buffers are permanent objects, there is no
special garbage collection-related trickery involved in these parent and
children pointers.  There should never be an indirect buffer whose
parent is also an indirect buffer.  If the user attempts to set up such
a situation using `make_indirect_buffer()', either an error should be
signaled or the parent of the indirect buffer should automatically
become the direct buffer that actually is responsible for the text.
Deleting a direct buffer should perhaps cause all of the indirect buffer
children to be deleted automatically.  There should be Lisp functions
for determining whether a buffer is direct or indirect, and other
functions for retrieving the parents, or the children of the buffer,
depending on which is appropriate.  (The scheme being described here is
similar to symbolic links.  Another possible scheme would be analogous
to hard links, and would make no distinction between direct and indirect
buffers.  In that case, the text of the buffer logically exists as an
object separate from the buffer itself and only goes away when the last
buffer pointing to this text is deleted.)

   Other than keeping track of parent and child pointer, the only
remaining thing required to implement indirect buffers is to ensure
that changes to the text of the buffer trigger the same sorts of effect
in all the buffers that share that text.  Luckily there are only three
functions in XEmacs that actually make changes to the text of the
buffer, and they are all located in the file `insdel.c'.

   These three functions are called `buffer_insert_string_1()',
`buffer_delete_range()', and `buffer_replace_char()'.  All of the
subfunctions called by these functions are also in `insdel.c'.

   The first thing that each of these three functions needs to do is
check to see if its buffer argument is an indirect buffer, and if so,
convert it to the indirect buffer's parent.  Once that is done, the
functions need to be modified so that all of the things they do, other
than actually changing the buffers text, such as calling
before-change-functions and after-change-functions, and updating extents
and markers, need to be done over all of the buffers that are indirect
children of the buffers being modified; as well as, of course, for the
buffer itself.  Each step in the process needs to be iterated for all of
the buffers in question before proceeding to the next step.  For
example, in `buffer_insert_string_1()', `prepare_to_modify_buffer()'
needs to be called in turn, for all of the buffers sharing the text
being modified.  Then the text itself is modified, then
`insert_invalidate_line_number_cache()' is called for all of the
buffers, then `record_insert()' is called for all of the buffers, etc.
Essentially, the operation is being done on all of the buffers in
parallel, rather than each buffer being processed in series.  This is
necessary because many of the steps can quit or call Lisp code and each
step depends on the previous step, and some steps are done only once,
rather than on each buffer.  I imagine it would be significantly easier
to implement this, if a macro were created for iterating over a buffer,
and then all of the indirect children of that buffer.


File: internals.info,  Node: Old Future Work -- Improvements in support for non-ASCII (European) keysyms under X,  Next: Old Future Work -- RTF Clipboard Support,  Prev: Old Future Work -- Indirect Buffers,  Up: Old Future Work

46.3 Old Future Work - Improvements in support for non-ASCII (European) keysyms under X
=======================================================================================

Author: Martin Buchholz (mailto:martin@xemacs.org)

   If a user has a keyboard with known standard non-ASCII character
equivalents, typically for European users, then Emacs' default binding
should be self-insert-command, with the obvious character inserted.
For example, if a user has a keyboard with

   xmodmap -e "keycode 54 = scaron"

   then pressing that key on the keyboard will insert the (Latin-2)
character corresponding to "scaron" into the buffer.

   Note: Emacs 20.6 does NOTHING when pressing such a key (not even an
error), i.e. even (read-event) ignores this key, which means it can't
even be bound to anything by a user trying to customize it.

   This is implemented by maintaining a table of translations between
all the known X keysym names and the corresponding (charset, octet)
pairs.

        For every key on the keyboard that has a known character
     correspondence,    we define the character-of-keysym property of
     the keysym, and make the    default binding for the key be
     self-insert-command.

     The following magic is basically intimate knowledge of
     X11/keysymdef.h.     The keysym mappings defined by X11 are based
     on the iso8859 standards,    except for Cyrillic and Greek.

     In a non-Mule world, a user can still have a multi-lingual editor,
     by doing    (set-face-font "...-iso8859-2" (current-buffer))
     for all their Latin-2 buffers, etc.


File: internals.info,  Node: Old Future Work -- RTF Clipboard Support,  Next: Old Future Work -- xemacs.org Mailing Address Changes,  Prev: Old Future Work -- Improvements in support for non-ASCII (European) keysyms under X,  Up: Old Future Work

46.4 Old Future Work - RTF Clipboard Support
============================================

Author: Ben Wing (mailto:ben@xemacs.org)

   in fact, i merged the windows stuff with the already-existing
generic code.

   what i'd like to see is something like this:

  1. The current function

          (defun own-selection (data &optional type append)

     should become

          (defun own-selection (data &optional type how-to-add data-type)

     where data-type is the mswindows format, and how-to-add is

          'replace-all or nil -- remove data for all formats
          'replace-existing -- remove data for DATA-TYPE, but leave other formats alone
          'append or t -- append data to existing data in DATA-TYPE, and leave other
          formats alone

  2. the function

          (get-selection &optional TYPE DATA-TYPE)

     already has a data-type so you don't need to change it.

  3. the existing function

          (selection-exists-p &optional SELECTION DEVICE)

     should become

          (selection-exists-p &optional SELECTION DEVICE DATA-TYPE)

  4. a new function

          (register-selection-data-type DATA-TYPE)

     like your mswindows-register-clipboard-format.

  5. there's already a selection-converter-alist, but that's only for
     data out.  you should alias it to selection-conversion-out-alist,
     and create selection-conversion-in-alist.  these alists contain
     entries for CF_TEXT, which handles CR/LF conversion, and rtf,
     which does rtf in/out conversion - no need for separate functions
     to do this.

     this may seem daunting, but it's much less hard to add stuff like
     this than it seems, and i and others will certainly give you lots
     of support if you run into problems.  it would be way cool to have
     a more powerful clipboard mechanism in XEmacs.


File: internals.info,  Node: Old Future Work -- xemacs.org Mailing Address Changes,  Next: Old Future Work -- Lisp callbacks from critical areas of the C code,  Prev: Old Future Work -- RTF Clipboard Support,  Up: Old Future Work

46.5 Old Future Work - xemacs.org Mailing Address Changes
=========================================================

Author: Ben Wing (mailto:ben@xemacs.org)

Personal addresses
------------------

  1.  Everyone who is contributing or has ever contributed code to the
     XEmacs core, or to any of the packages archived at xemacs.org,
     even if they don't actually have an account on any machine at
     xemacs.org. In fact, all of these people should have two mailing
     addresses at xemacs.org, one of which is their actual login name
     (or potential login name if they were ever to have an account),
     and the other one is in the form of first name/last name, similar
     to the way things are done at Sun.  For example, Martin would have
     two addresses at xemacs.org, `martin@xemacs.org', and
     `martin.buchholz@xemacs.org', with the latter one simply being an
     alias for the former.  The idea is that in all cases, if you simply
     know the name of any past or present contributor to XEmacs, and
     you want to mail them, you will know immediately how to do this
     without having to do any complicated searching on the Web or in
     XEmacs documentation.

  2.  Furthermore, I think that all of the email addresses mentioned
     anywhere in the XEmacs source code or documentation should be
     changed to be the corresponding ones at xemacs.org, instead of any
     other email addresses that any contributors might have.

  3.  All the places in the source code where a contributor's name is
     mentioned, but no email addressed is attached, should be found,
     and the correct xemacs.org address should be attached.

  4.  The alias file mapping people's addresses at xemacs.org to their
     actual addresses elsewhere (in the case, as will be true for the
     majority of addresses, where the contributor does not actually
     have an account at xemacs.org, but simply a forwarding pointer),
     should be viewable on the xemacs.org web site through a CGI script
     that reads the alias file and turns it into an HTML table.


Package addresses
-----------------

I also think that for every package archived at xemacs.org, there should
be three corresponding email addresses at xemacs.org.  For example,
consider a package such as `lazy-shot'.  The addresses associated with
this package would be:

`lazy-shot@xemacs.org'
     This is a discussion mailing list about the `lazy-shot' package,
     and it should be controlled by Majordomo in the standard fashion.

`lazy-shot-patches@xemacs.org'
     This is where patches to the `lazy-shot' package are set.  This
     should go to various people who are interested in such patches.
     For example, the maintainer of `lazy-shot', perhaps the maintainer
     of XEmacs itself, and probably to other people who have
     volunteered to do code review for this package, or for a larger
     group of packages that this package is in.  Perhaps this list
     should also be maintained by Majordomo.

`lazy-shot-maintainer@xemacs.org'
     This address is for mailing the maintainer directly.  It is
     possible that this will go to more than one person.  This would
     particularly be the case, for example, if the maintainer is
     dormant or does not appear very responsive to patches.  In this
     case, the address would also point to someone like Steve, who is
     acting in the maintainer's stead, and who will himself apply
     patches or make other changes to the package as maintained in the
     CVS archive on xemacs.org.

   It may take a bit of work to track down the current addresses for the
various package maintainers, and may in general seem like a lot of work
to set up all of these mail addresses, but I think it's very important
to make it as easy as possible for random XEmacs users to be able to
submit patches and report bugs in an orderly fashion.  The general idea
that I'm striving for is to create as much momentum as possible in the
XEmacs development community, and I think having the system of mail
addresses set up will make it much easier for this momentum to be built
up and to remain.

   Ben Wing (../../www.666.com/ben/default.htm)


File: internals.info,  Node: Old Future Work -- Lisp callbacks from critical areas of the C code,  Next: Old Future Work -- Everything should obey duplicable extents,  Prev: Old Future Work -- xemacs.org Mailing Address Changes,  Up: Old Future Work

46.6 Old Future Work - Lisp callbacks from critical areas of the C code
=======================================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   There are many places in the XEmacs C code where Lisp functions are
called, usually because the Lisp function is acting as a callback,
hook, process filter, or the like.  The lisp code is often called in
places where some lisp operations are dangerous.  Currently there are a
lot of ad-hoc schemes implemented to try to prevent these dangerous
operations from causing problems.  I've added a lot of them myself, for
example, the `call*_trapping_errors()' functions.  Other places, such
as the pre-gc- and post-gc-hooks, do their own ad hoc processing.  I'm
proposing a scheme that would generalize all of this ad hoc code and
allow Lisp code to be called in all sorts of sensitive areas of the C
code, including even within redisplay.

   Basically, we define a set of operations that are disallowable
because they are dangerous.  We essentially assign a bit flag to all of
these operations.  Whenever any sensitive C code wants to call Lisp
code, instead of using the standard call* functions, it uses a new set
of functions, call*_critical, which takes an extra parameter, which is a
bit mask specifying the set of operations which are disallowed.  The
basic operations of these functions is simply to set a global variable
corresponding to the bit mask (more specifically, the functions store
the previous value of this global variable in an unwind_protect, and
use bitwise-or to combine the previous value with the new bit mask that
was passed in).  (Actually, we should first implement a slightly lower
level function which is called `enter_sensitive_code_section()', which
simply sets up the global variable and the `unwind_protect()', and
returns a `specbind()' value, but doesn't actually call any Lisp code.
There is a corresponding function `exit_sensitive_code_section()', which
takes the specbind value as an argument, and unwinds the
unwind_protect.  The call*_sensitive functions are trivially
implemented in terms of these lower level functions.)

   Corresponding to each of these entries is the C name of the bit flag.

   The sets of dangerous operations which can be prohibited are:

`OPERATION_GC_PROHIBITED'
     garbage collection.  When this flag is set, and the garbage
     collection threshold is reached, garbage collection simply doesn't
     happen.  It will happen at the next opportunity that it is allowed.
     Similarly, explicitly calling the Lisp function garbage-collect
     simply does nothing.

`OPERATION_CATCH_ERRORS'
     signalling an error.  When `enter_sensitive_code_section()' is
     called, with the bit flag corresponding to this prohibited
     operation.  When this bit flag is passed to
     `enter_sensitive_code_section()', a catch is set up which catches
     all errors, signals a warning with `warn_when_safe()', and then
     simply continues.  This is exactly the same behavior you now get
     with the `call_*_trapping_errors()' functions.  (there should also
     be some way of specifying a warning level and class here, similar
     to the `call_*_trapping_errors()' functions.  This is not
     completely important, however, because a standard warning level
     and class could simply be chosen.)

`OPERATION_NO_UNSAFE_OBJECT_DELETION'
     This flag prohibits deletion of any permanent object (i.e. any
     object that does not automatically disappear when created, such as
     buffers, frames, devices, windows, etc...) unless they were created
     after this bit flag was set.  This would be implemented using a
     list which stores all of the permanent objects created after this
     bit flag was set.  This list is reset to its previous value when
     the call to `exit_sensitive_code_section()' occurs.  The motivation
     here is to allow Lisp callbacks to create their own temporary
     buffers or frames, and later delete them, but not allow any other
     permanent objects to be deleted, because C code might be working
     with them, and not expect them to change.

`OPERATION_NO_BUFFER_MODIFICATION'
     This flag disallows modifications to the text, extent or any other
     properties of any buffers except those created after this flag was
     set, just like in the previous entry.

`OPERATION_NO_REDISPLAY'
     This bit flag inhibits any redisplay-related operations from
     happening, more specifically, any entry into the redisplay-related
     code.  This includes, for example, the Lisp functions sit-for,
     force-redisplay, force-cursor-redisplay, window-end with certain
     arguments to it, and various other functions. When this flag is
     set, instead of entering the redisplay code, the calling function
     should simply make sure not to enter the redisplay code, (for
     example, in the case of window-end), or postpone the redisplay
     until such a time when it's safe (for example, with sit-for and
     force-redisplay).

`OPERATION_NO_REDISPLAY_SETTINGS_CHANGE'
     This flag prohibits any modifications to faces, glyphs, specifiers,
     extents, or any other settings that will affect the way that any
     window is displayed.

   The idea here is that it will finally be safe to call Lisp code from
nearly any part of the C code, simply by setting any combination of
restricted operation bit flags.  This even includes from within
redisplay. (in such a case, all of the bit flags need to be set).  The
reason that I thought of this is that some coding system translations
might cause Lisp code to be invoked and C code often invokes these
translations in sensitive places.


File: internals.info,  Node: Old Future Work -- Everything should obey duplicable extents,  Next: Old Future Work -- Display Tables,  Prev: Old Future Work -- Lisp callbacks from critical areas of the C code,  Up: Old Future Work

46.7 Old Future Work - Everything should obey duplicable extents
================================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   A lot of functions don't properly track duplicable extents.  For
example, the `concat' function does, but the `format' function does
not, and extents in keymap prompts are not displayed either.  All of
the functions that generate strings or string-like entities should
track the extents that are associated with the strings.  Currently this
is difficult because there is no general mechanism implemented for doing
this.  I propose such a general mechanism, which would not be hard to
implement, and would be easy to use in other functions that build up
strings.

   The basic idea is that we create a C structure that is analogous to a
Lisp string in that it contains string data and lists of extents for
that data.  Unlike standard Lisp strings, however, this structure (let's
call it `lisp_string_struct') can be incrementally updated and its
allocation is handled explicitly so that no garbage is generated.  (This
is important for example, in the event-handling code which would want to
use this structure, but needs to not generate any garbage for efficiency
reasons).  Both the string data and the list of extents in this string
are handled using dynarrs so that it is easy to incrementally update
this structure.  Functions should exist to create and destroy instances
of `lisp_string_struct' to generate a Lisp string from a
`lisp_string_struct' and vice-versa to append a sub-string of a Lisp
string to a `lisp_string_struct', to just append characters to a
`lisp_string_struct', etc.  The only thing possibly tricky about
implementing these functions is implementing the copying of extents from
a Lisp string into a `lisp_string_struct'.  However, there is already a
function `copy_string_extents()' that does basically this exact thing,
and it should be easy to create a modified version of this function.


File: internals.info,  Node: Old Future Work -- Display Tables,  Prev: Old Future Work -- Everything should obey duplicable extents,  Up: Old Future Work

46.8 Old Future Work - Display Tables
=====================================

Author: Ben Wing (mailto:ben@xemacs.org)

   It would also be really nice if you could specify that the characters
come out in hex instead of in octal.  Mule does that by adding a
`ctl-hexa' variable similar to `ctl-arrow', but that's bogus - we need
a more general solution.  I think you need to extend the concept of
display tables into a more general conversion mechanism.  Ideally you
could specify a Lisp function that converts characters, but this
violates the Second Golden Rule and besides would make things way way
way way slow.

   So instead, we extend the display-table concept, which was
historically limited to 256-byte vectors, to one of the following:

  1. A 256-entry vector, for backward compatibility;

  2. char-table, mapping characters to values;

  3. range-table, mapping ranges of characters to values;

  4. a list of the above.

   The fourth option allows you to specify multiple display tables
instead of just one.  Each display table can specify conversions for
some characters and leave others unchanged.  The way the character gets
displayed is determined by the first display table with a binding for
that character.  This way, you could call a function
`enable-hex-display' that adds a hex display-table to the list of
display tables for the current buffer.

   Also, we extend the concept of "mapping" to include a printf-like
spec.  Thus you can make all extended characters show up as hex with a
display table like this:

         #s(range-table data ((256 524288) (format "%x")))

   Since more than one display table is possible, you have great
flexibility in mapping ranges of characters.

   WARN

