This is ../info/internals.info, produced by makeinfo version 4.8 from
internals/internals.texi.

INFO-DIR-SECTION XEmacs Editor
START-INFO-DIR-ENTRY
* Internals: (internals).       XEmacs Internals Manual.
END-INFO-DIR-ENTRY

   Edition History:

   Created November 1995 (?) by Ben Wing.  XEmacs Internals Manual
Version 1.0, March, 1996.  XEmacs Internals Manual Version 1.1, March,
1997.  XEmacs Internals Manual Version 1.4, March, 2001.  XEmacs
Internals Manual Version 21.5, October, 2004.

   Copyright (C) 1992 - 2004 Ben Wing.  Copyright (C) 1996, 1997 Sun
Microsystems.  Copyright (C) 1994 - 1998, 2002, 2003 Free Software
Foundation.  Copyright (C) 1994, 1995 Board of Trustees, University of
Illinois.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that the
entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the Foundation.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided also
that the section entitled "GNU General Public License" is included
exactly as in the original, and provided that the entire resulting
derived work is distributed under the terms of a permission notice
identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that the section entitled "GNU General Public License"
may be included in a translation approved by the Free Software
Foundation instead of in the original English.


File: internals.info,  Node: Control-G (Quit) Checking,  Next: Profiling,  Prev: Signal Handling,  Up: Asynchronous Events; Quit Checking

38.2 Control-G (Quit) Checking
==============================

_Note_: The code to handle QUIT is divided between `lisp.h' and
`signal.c'.  There is also some special-case code in the async timer
code in `event-stream.c' to notice when the poll-for-quit (and
poll-for-sigchld) timers have gone off.

   Here's an overview of how this convoluted stuff works:

  1.  Scattered throughout the XEmacs core code are calls to the macro
     QUIT; This macro checks to see whether a `C-g' has recently been
     pressed and not yet handled, and if so, it handles the `C-g' by
     calling `signal_quit()', which invokes the standard `Fsignal()'
     code, with the error being `Qquit'.  Lisp code can establish
     handlers for this (using `condition-case'), but normally there is
     no handler, and so execution is thrown back to the innermost
     enclosing event loop. (One of the things that happens when
     entering an event loop is that a `condition-case' is established
     that catches *all* calls to `signal', including this one.)

  2. How does the QUIT macro check to see whether `C-g' has been
     pressed; obviously this needs to be extremely fast.  Now for some
     history.  In early Lemacs as inherited from the FSF going back 15
     years or more, there was a great fondness for using SIGIO (which
     is sent whenever there is I/O available on a given socket, tty,
     etc.).  In fact, in GNU Emacs, perhaps even today, all reading of
     events from the X server occurs inside the SIGIO handler!  This is
     crazy, but not completely relevant.  What is relevant is that
     similar stuff happened inside the SIGIO handler for `C-g': it
     searched through all the pending (i.e. not yet delivered to XEmacs
     yet) X events for one that matched `C-g'.  When it saw a match, it
     set Vquit_flag to Qt.  On TTY's, `C-g' is actually mapped to be the
     interrupt character (i.e. it generates SIGINT), and XEmacs's
     handler for this signal sets Vquit_flag to Qt.  Then, sometime
     later after the signal handlers finished and a QUIT macro was
     called, the macro noticed the setting of `Vquit_flag' and used
     this as an indication to call `signal_quit()'.  What
     `signal_quit()' actually does is set `Vquit_flag' to Qnil (so that
     we won't get repeated interruptions from a single `C-g' press) and
     then calls the equivalent of (signal 'quit nil).

  3. Another complication is introduced in that Vquit_flag is actually
     exported to Lisp as `quit-flag'.  This allows users some level of
     control over whether and when `C-g' is processed as quit, esp. in
     combination with `inhibit-quit'.  This is another Lisp variable,
     and if set to non-nil, it inhibits `signal_quit()' from getting
     called, meaning that the `C-g' gets essentially ignored.  But not
     completely: Because the resetting of `quit-flag' happens only in
     `signal_quit()', which isn't getting called, the `C-g' press is
     still noticed, and as soon as `inhibit-quit' is set back to nil, a
     quit will be signalled at the next QUIT macro.  Thus, what
     `inhibit-quit' really does is defer quits until after the quit-
     inhibitted period.

  4. Another consideration, introduced by XEmacs, is critical quitting.
     If you press `Control-Shift-G' instead of just `C-g', `quit-flag'
     is set to `critical' instead of to t.  When QUIT processes this
     value, it *ignores* the value of `inhibit-quit'.  This allows you
     to quit even out of a quit-inhibitted section of code!
     Furthermore, when `signal_quit()' notices that it was invoked as a
     result of a critical quit, it automatically invokes the debugger
     (which otherwise would only happen when `debug-on-quit' is set to
     t).

  5. Well, I explained above about how `quit-flag' gets set correctly,
     but I began with a disclaimer stating that this was the old way of
     doing things.  What's done now?  Well, first of all, the SIGIO
     handler (which formerly checked all pending events to see if
     there's a `C-g') now does nothing but set a flag - or actually two
     flags, something_happened and quit_check_signal_happened.  There
     are two flags because the QUIT macro is now used for more than
     just handling QUIT; it's also used for running asynchronous
     timeout handlers that have recently expired, and perhaps other
     things.  The idea here is that the QUIT macros occur extremely
     often in the code, but only occur at places that are relatively
     safe - in particular, if an error occurs, nothing will get
     completely trashed.

  6. Now, let's look at QUIT again.

  7.  UNFINISHED.  Note, however, that as of the point when this
     comment got committed to CVS (mid-2001), the interaction between
     reading `C-g' as an event and processing it as QUIT was overhauled
     to (for the first time) be understandable and actually work
     correctly.  Now, the way things work is that if `C-g' is pressed
     while XEmacs is blocking at the top level, waiting for a user
     event, it will be read as an event; otherwise, it will cause QUIT.
     (This includes times when XEmacs is blocking, but not waiting for
     a user event, e.g. `accept-process-output' and
     `wait_delaying_user_events()'.)  Formerly, this was supposed to
     happen, but didn't always due to a bizarre and broken scheme,
     documented in `next_event_internal' like this:

          If we read a `C-g', then set `quit-flag' but do not discard
          the `C-g'.  The callers of `next_event_internal()' will do
          one of two things:

            1. set `Vquit_flag' to Qnil. (`next-event' does this.) This
               will cause the ^G to be treated as a normal keystroke.

            2. not change `Vquit_flag' but attempt to enqueue the ^G,
               at which point it will be discarded.  The next time QUIT
               is called, it will notice that `Vquit_flag' was set.

     This required weirdness in `enqueue_command_event_1' like this:

          put the event on the typeahead queue, unless the event is the
          quit char, in which case the `QUIT' which will occur on the
          next trip through this loop is all the processing we should
          do - leaving it on the queue would cause the quit to be
          processed twice.

     And further weirdness elsewhere, none of which made any sense, and
     didn't work, because (e.g.) it required that QUIT never happen
     anywhere inside `next_event_internal()' or any callers when `C-g'
     should be read as a user event, which was impossible to implement
     in practice.

     Now what we do is fairly simple.  Callers of
     `next_event_internal()' that want `C-g' read as a user event call
     `begin_dont_check_for_quit()'.  `next_event_internal()', when it
     gets a `C-g', simply sets `Vquit_flag' (just as when a `C-g' is
     detected during the operation of `QUIT' or `QUITP'), and then
     tries to `QUIT'.  This will fail if blocked by the previous call,
     at which point `next_event_internal()' will return the `C-g' as an
     event.  To unblock things, first set `Vquit_flag' to nil (it was
     set to t when the `C-g' was read, and if we don't reset it, the
     next call to `QUIT' will quit), and then `unbind_to()' the depth
     returned by `begin_dont_check_for_quit()'.  It makes no difference
     is `QUIT' is called a zillion times in `next_event_internal()' or
     anywhere else, because it's blocked and will never signal.

38.2.1 Reentrancy Problems due to QUIT Checking
-----------------------------------------------

Checking for QUIT can do quite a long of things - since it pumps the
event loop, this may cause arbitrary code to get executed, garbage
collection to happen. etc. (In fact, garbage collection cannot happen
because it is inhibited.) This has led to crashes when functions get
called reentrantly when not expecting it.  Example:

Crash - reentrant `re_match_2()'
--------------------------------

       /* dont_check_for_quit is set in three circumstances:

          (1) when we are in the process of changing the window
          configuration.  The frame might be in an inconsistent state,
          which will cause assertion failures if we check for QUIT.

          (2) when we are reading events, and want to read the C-g
          as an event.  The normal check for quit will discard the C-g,
          which would be bad.

          (3) when we're going down with a fatal error.  we're most likely
          in an inconsistent state, and we definitely don't want to be
          interrupted. */

       /* We should *not* conditionalize on Vinhibit_quit, or
          critical-quit (Control-Shift-G) won't work right. */

       /* WARNING: Even calling check_quit(), without actually dispatching
          a quit signal, can result in arbitrary Lisp code getting executed
          -- at least under Windows. (Not to mention obvious Lisp
          invocations like asynchronous timer callbacks.) Here's a sample
          stack trace to demonstrate:

      NTDLL! DbgBreakPoint@0 address 0x77f9eea9
     assert_failed(const char * 0x012d036c, int 4596, const char * 0x012d0354) line 3478
     re_match_2_internal(re_pattern_buffer * 0x012d6780, const unsigned char * 0x00000000, int 0, const unsigned char * 0x022f9328, int 34, int 0, re_registers * 0x012d53d0 search_regs, int 34) line 4596 + 41 bytes
     re_search_2(re_pattern_buffer * 0x012d6780, const char * 0x00000000, int 0, const char * 0x022f9328, int 34, int 0, int 34, re_registers * 0x012d53d0 search_regs, int 34) line 4269 + 37 bytes
     re_search(re_pattern_buffer * 0x012d6780, const char * 0x022f9328, int 34, int 0, int 34, re_registers * 0x012d53d0 search_regs) line 4031 + 37 bytes
     string_match_1(long 31222628, long 30282164, long 28377092, buffer * 0x022fde00, int 0) line 413 + 69 bytes
     Fstring_match(long 31222628, long 30282164, long 28377092, long 28377092) line 436 + 34 bytes
     Ffuncall(int 3, long * 0x008297f8) line 3488 + 168 bytes
     execute_optimized_program(const unsigned char * 0x020ddc50, int 6, long * 0x020ddf50) line 744 + 16 bytes
     funcall_compiled_function(long 34407748, int 1, long * 0x00829aec) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x00829ae8) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020ddc90, int 4, long * 0x020ddf90) line 744 + 16 bytes
     funcall_compiled_function(long 34407720, int 1, long * 0x00829e28) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x00829e24) line 3523 + 17 bytes
     mapcar1(long 15, long * 0x00829e48, long 34447820, long 34187868) line 2929 + 11 bytes
     Fmapcar(long 34447820, long 34187868) line 3035 + 21 bytes
     Ffuncall(int 3, long * 0x00829f20) line 3488 + 93 bytes
     execute_optimized_program(const unsigned char * 0x020c2b70, int 7, long * 0x020dd010) line 744 + 16 bytes
     funcall_compiled_function(long 34407580, int 2, long * 0x0082a210) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082a20c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020cf810, int 6, long * 0x020cfb10) line 744 + 16 bytes
     funcall_compiled_function(long 34407524, int 0, long * 0x0082a580) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082a57c) line 3523 + 17 bytes
     run_hook_with_args_in_buffer(buffer * 0x022fde00, int 1, long * 0x0082a57c, int 0) line 3980 + 13 bytes
     run_hook_with_args(int 1, long * 0x0082a57c, int 0) line 3993 + 23 bytes
     Frun_hooks(int 1, long * 0x0082a57c) line 3847 + 19 bytes
     run_hook(long 34447484) line 4094 + 11 bytes
     unsafe_handle_wm_initmenu_1(frame * 0x01dbb000) line 736 + 11 bytes
     unsafe_handle_wm_initmenu(long 28377092) line 807 + 11 bytes
     condition_case_1(long 28377116, long (long)* 0x0101c827 unsafe_handle_wm_initmenu(long), long 28377092, long (long, long)* 0x01005fa4 mswindows_modal_loop_error_handler(long, long), long 28377092) line 1692 + 7 bytes
     mswindows_protect_modal_loop(long (long)* 0x0101c827 unsafe_handle_wm_initmenu(long), long 28377092) line 1194 + 32 bytes
     mswindows_handle_wm_initmenu(HMENU__ * 0x00010199, frame * 0x01dbb000) line 826 + 17 bytes
     mswindows_wnd_proc(HWND__ * 0x000501da, unsigned int 278, unsigned int 65945, long 0) line 3089 + 31 bytes
     USER32! UserCallWinProc@20 + 24 bytes
     USER32! DispatchClientMessage@20 + 47 bytes
     USER32! __fnDWORD@4 + 34 bytes
     NTDLL! KiUserCallbackDispatcher@12 + 19 bytes
     USER32! DispatchClientMessage@20 address 0x77e163cc
     USER32! DefWindowProcW@16 + 34 bytes
     qxeDefWindowProc(HWND__ * 0x000501da, unsigned int 274, unsigned int 61696, long 98) line 1188 + 22 bytes
     mswindows_wnd_proc(HWND__ * 0x000501da, unsigned int 274, unsigned int 61696, long 98) line 3362 + 21 bytes
     USER32! UserCallWinProc@20 + 24 bytes
     USER32! DispatchClientMessage@20 + 47 bytes
     USER32! __fnDWORD@4 + 34 bytes
     NTDLL! KiUserCallbackDispatcher@12 + 19 bytes
     USER32! DispatchClientMessage@20 address 0x77e163cc
     USER32! DefWindowProcW@16 + 34 bytes
     qxeDefWindowProc(HWND__ * 0x000501da, unsigned int 262, unsigned int 98, long 540016641) line 1188 + 22 bytes
     mswindows_wnd_proc(HWND__ * 0x000501da, unsigned int 262, unsigned int 98, long 540016641) line 3362 + 21 bytes
     USER32! UserCallWinProc@20 + 24 bytes
     USER32! DispatchMessageWorker@8 + 244 bytes
     USER32! DispatchMessageW@4 + 11 bytes
     qxeDispatchMessage(const tagMSG * 0x0082c684 {msg=0x00000106 wp=0x00000062 lp=0x20300001}) line 989 + 10 bytes
     mswindows_drain_windows_queue() line 1345 + 9 bytes
     emacs_mswindows_quit_p() line 3947
     event_stream_quit_p() line 666
     check_quit() line 686
     check_what_happened() line 437
     re_match_2_internal(re_pattern_buffer * 0x012d5a18, const unsigned char * 0x00000000, int 0, const unsigned char * 0x02235000, int 23486, int 14645, re_registers * 0x012d53d0 search_regs, int 23486) line 4717 + 14 bytes
     re_search_2(re_pattern_buffer * 0x012d5a18, const char * 0x02235000, int 23486, const char * 0x0223b38e, int 0, int 14645, int 8841, re_registers * 0x012d53d0 search_regs, int 23486) line 4269 + 37 bytes
     search_buffer(buffer * 0x022fde00, long 29077572, long 13789, long 23487, long 1, int 1, long 28377092, long 28377092, int 0) line 1224 + 89 bytes
     search_command(long 29077572, long 46975, long 28377116, long 28377092, long 28377092, int 1, int 1, int 0) line 1054 + 151 bytes
     Fre_search_forward(long 29077572, long 46975, long 28377116, long 28377092, long 28377092) line 2147 + 31 bytes
     Ffuncall(int 4, long * 0x0082ceb0) line 3488 + 216 bytes
     execute_optimized_program(const unsigned char * 0x02047810, int 13, long * 0x02080c10) line 744 + 16 bytes
     funcall_compiled_function(long 34187208, int 3, long * 0x0082d1b8) line 516 + 53 bytes
     Ffuncall(int 4, long * 0x0082d1b4) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x01e96a10, int 6, long * 0x020ae510) line 744 + 16 bytes
     funcall_compiled_function(long 34186676, int 3, long * 0x0082d4a0) line 516 + 53 bytes
     Ffuncall(int 4, long * 0x0082d49c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x02156b50, int 4, long * 0x020c2db0) line 744 + 16 bytes
     funcall_compiled_function(long 34186564, int 2, long * 0x0082d780) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082d77c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x0082d964, int 3, long * 0x020c2d70) line 744 + 16 bytes
     Fbyte_code(long 29405156, long 34352480, long 7) line 2392 + 38 bytes
     Feval(long 34354440) line 3290 + 187 bytes
     condition_case_1(long 34354572, long (long)* 0x01087232 Feval(long), long 34354440, long (long, long)* 0x01084764 run_condition_case_handlers(long, long), long 28377092) line 1692 + 7 bytes
     condition_case_3(long 34354440, long 28377092, long 34354572) line 1779 + 27 bytes
     execute_rare_opcode(long * 0x0082dc7c, const unsigned char * 0x01b090af, int 143) line 1269 + 19 bytes
     execute_optimized_program(const unsigned char * 0x01b09090, int 6, long * 0x020ae590) line 654 + 17 bytes
     funcall_compiled_function(long 34186620, int 0, long * 0x0082df68) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082df64) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x02195470, int 1, long * 0x020c2df0) line 744 + 16 bytes
     funcall_compiled_function(long 34186508, int 0, long * 0x0082e23c) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082e238) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x01e5d410, int 6, long * 0x0207d410) line 744 + 16 bytes
     funcall_compiled_function(long 34186312, int 1, long * 0x0082e524) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x0082e520) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x02108fb0, int 2, long * 0x020c2e30) line 744 + 16 bytes
     funcall_compiled_function(long 34186340, int 0, long * 0x0082e7fc) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082e7f8) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020fe150, int 2, long * 0x01e6f510) line 744 + 16 bytes
     funcall_compiled_function(long 31008124, int 0, long * 0x0082ebd8) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082ebd4) line 3523 + 17 bytes
     run_hook_with_args_in_buffer(buffer * 0x022fde00, int 1, long * 0x0082ebd4, int 0) line 3980 + 13 bytes
     run_hook_with_args(int 1, long * 0x0082ebd4, int 0) line 3993 + 23 bytes
     Frun_hooks(int 1, long * 0x0082ebd4) line 3847 + 19 bytes
     Ffuncall(int 2, long * 0x0082ebd0) line 3509 + 14 bytes
     execute_optimized_program(const unsigned char * 0x01ef2210, int 5, long * 0x01da8e10) line 744 + 16 bytes
     funcall_compiled_function(long 31020440, int 2, long * 0x0082eeb8) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082eeb4) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x0082f09c, int 3, long * 0x01d89390) line 744 + 16 bytes
     Fbyte_code(long 31102388, long 30970752, long 7) line 2392 + 38 bytes
     Feval(long 31087568) line 3290 + 187 bytes
     condition_case_1(long 30961240, long (long)* 0x01087232 Feval(long), long 31087568, long (long, long)* 0x01084764 run_condition_case_handlers(long, long), long 28510180) line 1692 + 7 bytes
     condition_case_3(long 31087568, long 28510180, long 30961240) line 1779 + 27 bytes
     execute_rare_opcode(long * 0x0082f450, const unsigned char * 0x01ef23ec, int 143) line 1269 + 19 bytes
     execute_optimized_program(const unsigned char * 0x01ef2310, int 6, long * 0x01da8f10) line 654 + 17 bytes
     funcall_compiled_function(long 31020412, int 1, long * 0x0082f740) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x0082f73c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020fe650, int 3, long * 0x01d8c490) line 744 + 16 bytes
     funcall_compiled_function(long 31020020, int 2, long * 0x0082fa14) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082fa10) line 3523 + 17 bytes
     Fcall_interactively(long 29685180, long 28377092, long 28377092) line 1008 + 22 bytes
     Fcommand_execute(long 29685180, long 28377092, long 28377092) line 2929 + 17 bytes
     execute_command_event(command_builder * 0x01be1900, long 36626492) line 4048 + 25 bytes
     Fdispatch_event(long 36626492) line 4341 + 70 bytes
     Fcommand_loop_1() line 582 + 9 bytes
     command_loop_1(long 28377092) line 495
     condition_case_1(long 28377188, long (long)* 0x01064fb9 command_loop_1(long), long 28377092, long (long, long)* 0x010649d0 cmd_error(long, long), long 28377092) line 1692 + 7 bytes
     command_loop_3() line 256 + 35 bytes
     command_loop_2(long 28377092) line 269
     internal_catch(long 28457612, long (long)* 0x01064b20 command_loop_2(long), long 28377092, int * volatile 0x00000000) line 1317 + 7 bytes
     initial_command_loop(long 28377092) line 305 + 25 bytes
     STACK_TRACE_EYE_CATCHER(int 1, char * * 0x01b63ff0, char * * 0x01ca5300, int 0) line 2501
     main(int 1, char * * 0x01b63ff0, char * * 0x01ca5300) line 2938
     XEMACS! mainCRTStartup + 180 bytes
     _start() line 171
     KERNEL32! BaseProcessStart@4 + 115547 bytes

   [explain dont_check_for_quit() et al]


File: internals.info,  Node: Profiling,  Next: Asynchronous Timeouts,  Prev: Control-G (Quit) Checking,  Up: Asynchronous Events; Quit Checking

38.3 Profiling
==============

We implement our own profiling scheme so that we can determine things
like which Lisp functions are occupying the most time.  Any standard
OS-provided profiling works on C functions, which is not always that
useful - and inconvenient, since it requires compiling with profile
info and can't be retrieved dynamically, as XEmacs is running.

   The basic idea is simple.  We set a profiling timer using setitimer
(ITIMER_PROF), which generates a SIGPROF every so often.  (This runs not
in real time but rather when the process is executing or the system is
running on behalf of the process - at least, that is the case under
Unix.  Under MS Windows and Cygwin, there is no `setitimer()', so we
simulate it using multimedia timers, which run in real time.  To make
the results a bit more realistic, we ignore ticks that go off while
blocking on an event wait.  Note that Cygwin does provide a simulation
of `setitimer()', but it's in real time anyway, since Windows doesn't
provide a way to have process-time timers, and furthermore, it's broken,
so we don't use it.) When the signal goes off, we see what we're in, and
add 1 to the count associated with that function.

   We use Lisp hash tables to keep track of the profiling information.
Updating the timing information happens inside of a signal handler, and
so we can't rely on not being in the middle of Lisp allocation, garbage
collection, `malloc()', etc. This means we need to be careful to ensure
the profiling table has an entry for every Lisp function ahead of any
call to that handler, because otherwise the hash table may need to be
resized in the signal handler. This is achieved by creating a zero
entry in the hash table in `profile_record_about_to_call()' and
creating zero entries if needed for those functions already on the
backtrace frame within `start-profile'.

   In general: any entry that the sigprof handler puts into the table
comes from a backtrace frame (except "Processing Events at Top Level",
and there's only one of those).  Either that backtrace frame was added
when profiling was on (in which case `profile_record_about_to_call()'
was called and the breathing space updated), or when it was off - and in
this case, no such frames can have been added since the last time
`start-profile' was called, so when `start-profile' is called we make
sure there is sufficient breathing room to account for all entries
currently on the stack.

   Jan 1998: In addition to timing info, I have added code to remember
call counts of Lisp funcalls.  The `profile_increase_call_count()'
function is called from `Ffuncall()', and serves to add data to
Vcall_count_profile_table.  This mechanism is much simpler and
independent of the SIGPROF-driven one.  It uses the Lisp allocation
mechanism normally, since it is not called from a handler.  It may even
be useful to provide a way to turn on only one profiling mechanism, but
I haven't done so yet.  -hniksic

   Dec 2002: Total overhaul of the interface, making it sane and easier
to use. -ben

   Feb 2003: Lots of rewriting of the internal code.  Add
GC-consing-usage, total GC usage, and total timing to the information
tracked.  Track profiling overhead and allow the ability to have
internal sections (e.g. internal-external conversion, byte-char
conversion) that are treated like Lisp functions for the purpose of
profiling.  -ben

   BEWARE: If you are modifying this file, be *very* careful.  Correctly
implementing the "total" values is very tricky due to the possibility of
recursion and of functions already on the stack when starting to
profile/still on the stack when stopping.


File: internals.info,  Node: Asynchronous Timeouts,  Next: Exiting,  Prev: Profiling,  Up: Asynchronous Events; Quit Checking

38.4 Asynchronous Timeouts
==========================


File: internals.info,  Node: Exiting,  Prev: Asynchronous Timeouts,  Up: Asynchronous Events; Quit Checking

38.5 Exiting
============

Ben's capsule summary about expected and unexpected exits from XEmacs.

   Expected exits occur when the user directs XEmacs to exit, for
example by pressing the close button on the only frame in XEmacs, or by
typing `C-x C-c'.  This runs `save-buffers-kill-emacs', which saves any
necessary buffers, and then exits using the primitive `kill-emacs'.

   However, unexpected exits occur in a few different ways:

   * A memory access violation or other hardware-generated exception
     occurs.  This is the worst possible problem to deal with, because
     the fault can occur while XEmacs is in any state whatsoever, even
     quite unstable ones.  As a result, we need to be *extremely*
     careful what we do.

   * We are using one X display (or if we've used more, we've closed the
     others already), and some hardware or other problem happens and
     suddenly we've lost our connection to the display.  In this
     situation, things are not so dire as in the last one; our code
     itself isn't trashed, so we can continue execution as normal,
     after having set things up so that we can exit at the appropriate
     time.  Our exit still needs to be of the emergency nature; we have
     no displays, so any attempts to use them will fail.  We simply
     want to auto-save (the single most important thing to do during
     shut-down), do minimal cleanup of stuff that has an independent
     existence outside of XEmacs, and exit.

   Currently, both unexpected exit scenarios described above set
`preparing_for_armageddon' to indicate that nonessential and possibly
dangerous things should not be done, specifically:

   - no garbage collection.

   - no hooks are run.

   - no messages of any sort from autosaving.

   - autosaving tries harder, ignoring certain failures.

   - existing frames are not deleted.

   (Also, all places that set `preparing_for_armageddon' also set
`dont_check_for_quit'.  This happens separately because it's also
necessary to set other variables to make absolutely sure no quitting
happens.)

   In the first scenario above (the access violation), we also set
`fatal_error_in_progress'.  This causes more things to not happen:

   - assertion failures do not abort.

   - printing code does not do code conversion or gettext when printing
     to stdout/stderr.


File: internals.info,  Node: Lstreams,  Next: Subprocesses,  Prev: Asynchronous Events; Quit Checking,  Up: Top

39 Lstreams
***********

An "lstream" is an internal Lisp object that provides a generic
buffering stream implementation.  Conceptually, you send data to the
stream or read data from the stream, not caring what's on the other end
of the stream.  The other end could be another stream, a file
descriptor, a stdio stream, a fixed block of memory, a reallocating
block of memory, etc.  The main purpose of the stream is to provide a
standard interface and to do buffering.  Macros are defined to read or
write characters, so the calling functions do not have to worry about
blocking data together in order to achieve efficiency.

   Lstreams seem to be similar to Python 2.5 generators/coroutines.
David Beasley's tutorials on generators
(http://dabeaz.com/generators-uk/) and coroutines
(http://dabeaz.com/coroutines/) are an excellent resource.

* Menu:

* Creating an Lstream::         Creating an lstream object.
* Lstream Types::               Different sorts of things that are streamed.
* Lstream Functions::           Functions for working with lstreams.
* Lstream Methods::             Creating new lstream types.


File: internals.info,  Node: Creating an Lstream,  Next: Lstream Types,  Prev: Lstreams,  Up: Lstreams

39.1 Creating an Lstream
========================

Lstreams come in different types, depending on what is being interfaced
to.  Although the primitive for creating new lstreams is
`Lstream_new()', generally you do not call this directly.  Instead, you
call some type-specific creation function, which creates the lstream
and initializes it as appropriate for the particular type.

   All lstream creation functions take a MODE argument, specifying what
mode the lstream should be opened as.  This controls whether the
lstream is for input and output, and optionally whether data should be
blocked up in units of MULE characters.  Note that some types of
lstreams can only be opened for input; others only for output; and
others can be opened either way.  #### Richard Mlynarik thinks that
there should be a strict separation between input and output streams,
and he's probably right.

   MODE is a string, one of

`"r"'
     Open for reading.

`"w"'
     Open for writing.

`"rc"'
     Open for reading, but "read" never returns partial MULE characters.

`"wc"'
     Open for writing, but never writes partial MULE characters.


File: internals.info,  Node: Lstream Types,  Next: Lstream Functions,  Prev: Creating an Lstream,  Up: Lstreams

39.2 Lstream Types
==================

stdio

filedesc

lisp-string

fixed-buffer

resizing-buffer

dynarr

lisp-buffer

print

decoding

encoding


File: internals.info,  Node: Lstream Functions,  Next: Lstream Methods,  Prev: Lstream Types,  Up: Lstreams

39.3 Lstream Functions
======================

 -- Function: Lstream * Lstream_new (Lstream_implementation *IMP, const
          char *MODE)
     Allocate and return a new Lstream.  This function is not really
     meant to be called directly; rather, each stream type should
     provide its own stream creation function, which creates the stream
     and does any other necessary creation stuff (e.g. opening a file).

 -- Function: void Lstream_set_buffering (Lstream *LSTR,
          Lstream_buffering BUFFERING, int BUFFERING_SIZE)
     Change the buffering of a stream.  See `lstream.h'.  By default the
     buffering is `STREAM_BLOCK_BUFFERED'.

 -- Function: int Lstream_flush (Lstream *LSTR)
     Flush out any pending unwritten data in the stream.  Clear any
     buffered input data.  Returns 0 on success, -1 on error.

 -- Macro: int Lstream_putc (Lstream *STREAM, int C)
     Write out one byte to the stream.  This is a macro and so it is
     very efficient.  The C argument is only evaluated once but the
     STREAM argument is evaluated more than once.  Returns 0 on
     success, -1 on error.

 -- Macro: int Lstream_getc (Lstream *STREAM)
     Read one byte from the stream.  This is a macro and so it is very
     efficient.  The STREAM argument is evaluated more than once.
     Return value is -1 for EOF or error.

 -- Macro: void Lstream_ungetc (Lstream *STREAM, int C)
     Push one byte back onto the input queue.  This will be the next
     byte read from the stream.  Any number of bytes can be pushed back
     and will be read in the reverse order they were pushed back--most
     recent first. (This is necessary for consistency--if there are a
     number of bytes that have been unread and I read and unread a
     byte, it needs to be the first to be read again.) This is a macro
     and so it is very efficient.  The C argument is only evaluated
     once but the STREAM argument is evaluated more than once.

 -- Function: int Lstream_fputc (Lstream *STREAM, int C)
 -- Function: int Lstream_fgetc (Lstream *STREAM)
 -- Function: void Lstream_fungetc (Lstream *STREAM, int C)
     Function equivalents of the above macros.

 -- Function: Bytecount Lstream_read (Lstream *STREAM, void *DATA,
          Bytecount SIZE)
     Read SIZE bytes of DATA from the stream.  Return the number of
     bytes read.  0 means EOF. -1 means an error occurred and no bytes
     were read.

 -- Function: Bytecount Lstream_write (Lstream *STREAM, void *DATA,
          Bytecount SIZE)
     Write SIZE bytes of DATA to the stream.  Return the number of
     bytes written.  -1 means an error occurred and no bytes were
     written.

 -- Function: void Lstream_unread (Lstream *STREAM, void *DATA,
          Bytecount SIZE)
     Push back SIZE bytes of DATA onto the input queue.  The next call
     to `Lstream_read()' with the same size will read the same bytes
     back.  Note that this will be the case even if there is other
     pending unread data.

 -- Function: int Lstream_close (Lstream *STREAM)
     Close the stream.  All data will be flushed out.

 -- Function: void Lstream_reopen (Lstream *STREAM)
     Reopen a closed stream.  This enables I/O on it again.  This is not
     meant to be called except from a wrapper routine that reinitializes
     variables and such--the close routine may well have freed some
     necessary storage structures, for example.

 -- Function: void Lstream_rewind (Lstream *STREAM)
     Rewind the stream to the beginning.


File: internals.info,  Node: Lstream Methods,  Prev: Lstream Functions,  Up: Lstreams

39.4 Lstream Methods
====================

 -- Lstream Method: Bytecount reader (Lstream *STREAM, unsigned char
          *DATA, Bytecount SIZE)
     Read some data from the stream's end and store it into DATA, which
     can hold SIZE bytes.  Return the number of bytes read.  A return
     value of 0 means no bytes can be read at this time.  This may be
     because of an EOF, or because there is a granularity greater than
     one byte that the stream imposes on the returned data, and SIZE is
     less than this granularity. (This will happen frequently for
     streams that need to return whole characters, because
     `Lstream_read()' calls the reader function repeatedly until it has
     the number of bytes it wants or until 0 is returned.)  The lstream
     functions do not treat a 0 return as EOF or do anything special;
     however, the calling function will interpret any 0 it gets back as
     EOF.  This will normally not happen unless the caller calls
     `Lstream_read()' with a very small size.

     This function can be `NULL' if the stream is output-only.

 -- Lstream Method: Bytecount writer (Lstream *STREAM, const unsigned
          char *DATA, Bytecount SIZE)
     Send some data to the stream's end.  Data to be sent is in DATA
     and is SIZE bytes.  Return the number of bytes sent.  This
     function can send and return fewer bytes than is passed in; in that
     case, the function will just be called again until there is no
     data left or 0 is returned.  A return value of 0 means that no
     more data can be currently stored, but there is no error; the data
     will be squirreled away until the writer can accept data. (This is
     useful, e.g., if you're dealing with a non-blocking file
     descriptor and are getting `EWOULDBLOCK' errors.)  This function
     can be `NULL' if the stream is input-only.

 -- Lstream Method: int rewinder (Lstream *STREAM)
     Rewind the stream.  If this is `NULL', the stream is not seekable.

 -- Lstream Method: int seekable_p (Lstream *STREAM)
     Indicate whether this stream is seekable--i.e. it can be rewound.
     This method is ignored if the stream does not have a rewind
     method.  If this method is not present, the result is determined
     by whether a rewind method is present.

 -- Lstream Method: int flusher (Lstream *STREAM)
     Perform any additional operations necessary to flush the data in
     this stream.

 -- Lstream Method: int pseudo_closer (Lstream *STREAM)

 -- Lstream Method: int closer (Lstream *STREAM)
     Perform any additional operations necessary to close this stream
     down.  May be `NULL'.  This function is called when
     `Lstream_close()' is called or when the stream is
     garbage-collected.  When this function is called, all pending data
     in the stream will already have been written out.

 -- Lstream Method: Lisp_Object marker (Lisp_Object LSTREAM, void
          (*MARKFUN) (Lisp_Object))
     Mark this object for garbage collection.  Same semantics as a
     standard `Lisp_Object' marker.  This function can be `NULL'.


File: internals.info,  Node: Subprocesses,  Next: Interface to MS Windows,  Prev: Lstreams,  Up: Top

40 Subprocesses
***************

The fields of a process are:

`name'
     A string, the name of the process.

`command'
     A list containing the command arguments that were used to start
     this process.

`filter'
     A function used to accept output from the process instead of a
     buffer, or `nil'.

`sentinel'
     A function called whenever the process receives a signal, or `nil'.

`buffer'
     The associated buffer of the process.

`pid'
     An integer, the Unix process ID.

`childp'
     A flag, non-`nil' if this is really a child process.  It is `nil'
     for a network connection.

`mark'
     A marker indicating the position of the end of the last output
     from this process inserted into the buffer.  This is often but not
     always the end of the buffer.

`kill_without_query'
     If this is non-`nil', killing XEmacs while this process is still
     running does not ask for confirmation about killing the process.

`raw_status_low'
`raw_status_high'
     These two fields record 16 bits each of the process status
     returned by the `wait' system call.

`status'
     The process status, as `process-status' should return it.

`tick'
`update_tick'
     If these two fields are not equal, a change in the status of the
     process needs to be reported, either by running the sentinel or by
     inserting a message in the process buffer.

`pty_flag'
     Non-`nil' if communication with the subprocess uses a PTY; `nil'
     if it uses a pipe.

`infd'
     The file descriptor for input from the process.

`outfd'
     The file descriptor for output to the process.

`subtty'
     The file descriptor for the terminal that the subprocess is using.
     (On some systems, there is no need to record this, so the value is
     `-1'.)

`tty_name'
     The name of the terminal that the subprocess is using, or `nil' if
     it is using pipes.

* Menu:

* Ben's separate stderr notes::  Probably obsolete.


File: internals.info,  Node: Ben's separate stderr notes,  Prev: Subprocesses,  Up: Subprocesses

40.1 Ben's separate stderr notes (probably obsolete)
====================================================

This node contains some notes that Ben kept on his separate subprocess
workspace.  These notes probably describe changes and features that have
already been included in XEmacs 21.5; somebody should check and/or ask
Ben.

ben-separate-stderr-improved-error-trapping
===========================================

this is an old workspace, very close to being done, containing

   * subprocess stderr output can be read separately; needed to fully
     implement call-process with asynch. subprocesses.

   * huge improvements to the internal error-trapping routines (i.e. the
     routines that call Lisp code and trap errors); Lisp code can now be
     called from within redisplay.

   * cleanup and simplification of C-g handling; some things work now
     that never used to.

   * see the ChangeLogs in the workspace.


File: internals.info,  Node: Interface to MS Windows,  Next: Interface to the X Window System,  Prev: Subprocesses,  Up: Top

41 Interface to MS Windows
**************************

* Menu:

* Different kinds of Windows environments::
* Windows Build Flags::
* Windows I18N Introduction::
* Modules for Interfacing with MS Windows::
* CHANGES from 21.4-windows branch::  Probably obsolete.


File: internals.info,  Node: Different kinds of Windows environments,  Next: Windows Build Flags,  Prev: Interface to MS Windows,  Up: Interface to MS Windows

41.1 Different kinds of Windows environments
============================================

(a) operating system (OS) vs. window system vs. Win32 API vs. C runtime library (CRT) vs. and compiler
......................................................................................................

There are various Windows operating systems (Windows NT, 2000, XP, 95,
98, ME, etc.), which come in two basic classes: Windows NT (NT, 2000,
XP, and all future versions) and 9x (95, 98, ME).  9x-class operating
systems are a kind of hodgepodge of a 32-bit upper layer on top of a
16-bit MS-DOS-compatible lower layer.  NT-class operating systems are
written from the ground up as 32-bit (there are also 64-bit versions
available now), and provide many more features and much greater
stability, since there is full memory protection between all processes
and the between processes and the system.  NT-class operating systems
also provide emulation for DOS programs inside of a "sandbox" (i.e. a
walled-off environment in which one DOS program can screw up another
one, but there is theoretically no way for a DOS program to screw up the
OS itself).  From the perspective of XEmacs, the different between NT
and 9x is very important in Unicode support (not really provided under
9x - see `intl-win32.c') and subprocess creation, among other things.

   The operating system provides the framework for accessing files and
devices and running programs.  From the perspective of a program, the
operating system provides a set of services.  At the lowest level, the
way to call these services is dependent on the processor the OS is
running on, but a portable interface is provided to C programs through
functions called "system calls".  Under Windows, this interface is
called the Win32 API, and includes file-manipulation calls such as
`CreateFile()' and `ReadFile()', process-creation calls such as
`CreateProcess()', etc.

   This concept of system calls goes back to Unix, where similar
services are available but through routines with different, simpler
names, such as `open()', `read()', `fork()', `execve()', etc.  In
addition, Unix provides a higher layer of routines, called the C
Runtime Library (CRT), which provide higher-level, more convenient
versions of the same services (e.g.  "stream-oriented" file routines
such as `fopen()' and `fread()') as well as various other utility
functions, such as string-manipulation routines (e.g. `strcpy()' and
`strcmp()').

   For compatibility, a C Runtime Library (CRT) is also provided under
Windows, which provides a partial implementation of both the Unix CRT
and the Unix system-call API, implemented using the Win32 API.  The CRT
sources come with Visual C++ (VC++).  For example, under VC++ 6, look in
the CRT/SRC directory, e.g. for me (ben): /Program Files/Microsoft
Visual Studio/VC98/CRT/SRC. The CRT is provided using either MSVCRT
(dynamically linked) or `LIBC.LIB' (statically linked).

   The window system provides the framework for creating overlapped
windows and unifying signals provided by various devices (input devices
such as the keyboard and mouse, timers, etc.) into a single event queue
(or "message queue", under Windows).  Like the operating system, the
window system can be viewed from the perspective of a program as a set
of services provided by an API of function calls.  Under Windows,
window-system services are also available through the Win32 API, while
under UNIX the window system is typically a separate component (e.g. the
X Windowing System, aka X Windows or X11).  The term "GUI" ("graphical
user interface") is often used to refer to the services provided by the
window system, or to a windowing interface provided by a program.

   The Win32 API is implemented by various dynamic libraries, or DLL's.
The most important are KERNEL32, USER32, and GDI32.  KERNEL32 implements
the basic file-system and process services.  USER32 implements the
fundamental window-system services such as creating windows and handling
messages.  GDI32 implements higher-level drawing capabilities - fonts,
colors, lines, etc.

   C programs are compiled into executables using a compiler.  Under
Unix, a compiler usually comes as part of the operating system, but not
under Windows, where the compiler is a separate product.  Even under
Unix, people often install their own compilers, such as gcc.  Under
Windows, the Microsoft-standard compiler is Visual C++ (VC++).

   It is possible to provide an emulation of any API using any other, as
long as the underlying API provides the suitable functionality.  This is
what Cygwin (www.cygwin.com) does.  It provides a fairly complete POSIX
emulation layer (POSIX is a government standard for Unix behavior) on
top of MS Windows - in particular, providing the file-system, process,
tty, and signal semantics that are part of a modern, standard Unix
operating system.  Cygwin does this using its own DLL, `cygwin1.dll',
which makes calls to the Win32 API services in `kernel32.dll'.  Cygwin
also provides its own implementation of the C runtime library, called
`newlib' (`libcygwin.a'; `libc.a' and `libm.a' are symlinked to it),
which is implemented on top of the Unix system calls provided in
`cygwin1.dll'.  In addition, Cygwin provides static import libraries
that give you direct access to the Win32 API - XEmacs uses this to
provide GUI support under Cygwin.  Cygwin provides a version of GCC
(the GNU Project C compiler) that is set up to automatically link with
the appropriate Cygwin libraries.  Cygwin also provides, as optional
components, pre-compiled binaries for a great number of open-source
programs compiled under the Cygwin environment.  This includes all of
the standard Unix file-system, text-manipulation, development,
networking, database, etc. utilities, a version of X Windows that uses
the Win32 API underlyingly (see below), and compilations of nearly all
other common open-source packages (Apache, TeX, [X]Emacs, Ghostscript,
GTK, ImageMagick, etc.).

   Similarly, you can emulate the functionality of X Windows using the
Win32 component of the Win32 API.  Cygwin provides a package to do this,
from the XFree86 project.  Other versions of X under Windows also exist,
such as the MicroImages MI/X server.  Each version potentially can come
comes with its own header and library files, allowing you to compile
X-Windows programs.

   All of these different operating system and emulation layers can make
for a fair amount of confusion, so:

(b) CRT is not the same as VC++
...............................

Note that the CRT is *NOT* (completely) part of VC++.  True, if you link
statically, the CRT (in the form of `LIBC.LIB', which comes with VC++)
will be inserted into the executable (.EXE), but otherwise the CRT will
be separate.  The dynamic version of the CRT is provided by `MSVCRT.DLL'
(or `MSVCRTD.DLL', for debugging), which comes with Windows.  Hence,
it's possible to use a different compiler and still link with MSVCRT -
which is exactly what MinGW does.

(c) CRT is not the same as the Win32 API
........................................

Note also that the CRT is totally separate from the Win32 API.  They
provide different functions and are implemented in different DLL's.
They are also different levels - the CRT is implemented on top of
Win32.  Sometimes the CRT and Win32 both have their own versions of
similar concepts, such as locales.  These are typically maintained
separately, and can get out of sync.  Do not assume that changing a
setting in the CRT will have any effect on Win32 API routines using a
similar concept unless the CRT docs specifically say so.  Do not assume
that behavior described for CRT functions applies to Win32 API or
vice-versa.  Note also that the CRT knows about and is implemented on
top of the Win32 API, while the Win32 API knows nothing about the CRT.

(d) MinGW is not the same as Cygwin
...................................

As described in (b), Microsoft's version of the CRT (`MSVCRT.DLL') is
provided as part of Windows, separate from VC++, which must be
purchased.  Hence, it is possible to write MSVCRT to provide CRT
services without using VC++.  This is what MinGW (www.mingw.org) does -
it is a port of GCC that will use MSVCRT.  The reason one might want to
do this is (a) it is free, and (b) it does not require a separately
installed DLL, as Cygwin does. (#### Maybe MinGW targets CRTDLL, not
MSVCRT?  If so, what is CRTDLL, and how does it differ from MSVCRT and
`LIBC.LIB'?) Primarily, what MinGW provides is patches to GCC (now
integrated into the standard distribution) and its own header files and
import libraries that are compatible with MSVCRT.  The best way to think
of MinGW is as simply another Windows compiler, like how there used to
be Microsoft and Borland compilers.  Because MinGW programs use all the
same libraries as VC++ programs, and hence the same services are
available, programs that compile under VC++ should compile under MinGW
with very little change, whereas programs that compile under Cygwin will
look quite different.

   The confusion between MinGW and Cygwin is the confusion between the
environment that a compiler runs under and the target environment of a
program, i.e. the environment that a program is compiled to run under.
It's theoretically possible, for example, to compile a program under
Windows and generate a binary that can only be run under Linux, or
vice-versa - or, for that matter, to use Windows, running on an Intel
machine to write and a compile a program that will run on the Mac OS,
running on a PowerPC machine.  This is called cross-compiling, and while
it may seem rather esoteric, it is quite normal when you want to
generate a program for a machine that you cannot develop on - for
example, a program that will run on a Palm Pilot.  Originally, this is
how MinGW worked - you needed to run GCC under a Cygwin environment and
give it appropriate flags, telling it to use the MinGW headers and
target `MSVCRT.DLL' rather than `CYGWIN1.DLL'. (In fact, Cygwin
standardly comes with MinGW's header files.) This was because GCC was
written with Unix in mind and relied on a large amount of Unix-specific
functionality.  To port GCC to Windows without using a POSIX emulation
layer would mean a lot of rewriting of GCC.  Eventually, however, this
was done, and it GCC was itself compiled using MinGW.  The result is
that currently you can develop MinGW applications either under Cygwin
or under native Windows.

(e) Operating system is not the same as window system
.....................................................

As per the above discussion, we can use either Native Windows (the OS
part of Win32 provided by `KERNEL32.DLL' and the Windows CRT as
provided by MSVCRT or CLL) or Cygwin to provide operating-system
functionality, and we can use either Native Windows (the windowing part
of Win32 as provided by `USER32.DLL' and `GDI32.DLL') or X11 to provide
window-system functionality.  This gives us four possible build
environments.  It's currently possible to build XEmacs with at least
three of these combinations - as far as I know native + X11 is no
longer supported, although it used to be (support used to exist in
`xemacs.mak' for linking with some X11 libraries available from
somewhere, but it was bit-rotting and you could always use Cygwin; ####
what happens if we try to compile with MinGW, native OS + X11?).  This
may still seem confusing, so:

Native OS + native windowing
     We call `CreateProcess()' to run subprocesses (`process-nt.c'),
     and `CreateWindowEx()' to create a top-level window
     (`frame-msw.c').  We use `nt/xemacs.mak' to compile with VC++,
     linking with the Windows CRT (`MSVCRT.DLL' or `LIBC.LIB') and with
     the various Win32 DLL's (`KERNEL32.DLL', `USER32.DLL',
     `GDI32.DLL'); or we use `src/Makefile[.in.in]' to compile with
     GCC, telling it (e.g. -mno-cygwin, see `s/mingw32.h') to use MinGW
     (which will end up linking with `MSVCRT.DLL'), and linking GCC
     with -lshell32 -lgdi32 -luser32 etc. (see `configure.in').

Cygwin + native windowing
     We call `fork()'/`execve()' to run subprocesses
     (`process-unix.c'), and `CreateWindowEx()' to create a top-level
     window (`frame-msw.c').  We use `src/Makefile[in.in]' to compile
     with GCC (it will end up linking with `CYGWIN1.DLL') and link GCC
     with -lshell32 -lgdi32 -luser32 etc. (see `configure.in').

Cygwin + X11
     We call `fork()'/`execve()' to run subprocesses
     (`process-unix.c'), and `XtCreatePopupShell()' to create a
     top-level window (`frame-x.c').  We use `src/Makefile[.in.in]' to
     compile with GCC (it will end up linking with `CYGWIN1.DLL') and
     link GCC with -lXt, -lX11, etc. (see `configure.in').

     Finally, if native OS + X11 were possible, it might look something
     like

[Native OS + X11]
     We call `CreateProcess()' to run subprocesses (`process-nt.c'),
     and `XtCreatePopupShell()' to create a top-level window
     (`frame-x.c').  We use `nt/xemacs.mak' to compile with VC++,
     linking with the Windows CRT (`MSVCRT.DLL' or `LIBC.LIB') and with
     the various X11 DLL's (`XT.DLL', `XLIB.DLL', etc.); or we use
     `src/Makefile[.in.in]' to compile with GCC, telling it (e.g.
     -mno-cygwin, see `s/mingw32.h') to use MinGW (which will end up
     linking with `MSVCRT.DLL'), and linking GCC with -lXt, -lX11, etc.
     (see `configure.in').

   One of the reasons that we maintain the ability to build under Cygwin
and X11 on Windows, when we have native support, is that it allows
Windows compilers to test under a Unix-like environment.


File: internals.info,  Node: Windows Build Flags,  Next: Windows I18N Introduction,  Prev: Different kinds of Windows environments,  Up: Interface to MS Windows

41.2 Windows Build Flags
========================

`CYGWIN'
     for Cygwin-only stuff.

`WIN32_NATIVE'
     Win32 native OS-level stuff (files, process, etc.).  Applies
     whenever linking against the native C libraries - i.e.  all
     compilations with VC++ and with MINGW, but never Cygwin.

`HAVE_X_WINDOWS'
     for X Windows (regardless of whether under MS Win)

`HAVE_MS_WINDOWS'
     MS Windows native windowing system (anything related to the
     appearance of the graphical screen).  May or may not apply to any
     of VC++, MINGW, Cygwin.

   Finally, there's also the MINGW build environment, which uses GCC
(similar to Cygwin), but native MS Windows libraries rather than a
POSIX emulation layer (the Cygwin approach).  This environment defines
WIN32_NATIVE, but also defines MINGW, which is used mostly because uses
its own include files (related to Cygwin), which have a few things
messed up.

   Formerly, we had a whole host of flags.  Here's the conversion, for
porting code from GNU Emacs and such:

Old Constant       New Constant
---------------------------------------------------------------
`WINDOWSNT'        `WIN32_NATIVE'
`WIN32'            `WIN32_NATIVE'
`_WIN32'           `WIN32_NATIVE'
`HAVE_WIN32'       `WIN32_NATIVE'
`DOS_NT'           `WIN32_NATIVE'
`HAVE_NTGUI'       `WIN32_NATIVE', unless it ends up already bracketed
                   by this
`HAVE_FACES'       always true
`MSDOS'            determine whether this code is really specific to
                   MS-DOS (and not Windows - e.g. DJGPP code); if so,
                   delete the code; otherwise, convert to `WIN32_NATIVE'
                   (we do not support MS-DOS w/DOS Extender under XEmacs)
`__CYGWIN__'       `CYGWIN'
`__CYGWIN32__'     `CYGWIN'
`__MINGW32__'      `MINGW'


File: internals.info,  Node: Windows I18N Introduction,  Next: Modules for Interfacing with MS Windows,  Prev: Windows Build Flags,  Up: Interface to MS Windows

41.3 Windows I18N Introduction
==============================

*Abstract:* This page provides an overview of the aspects of the Win32
internationalization API that are relevant to XEmacs, including the
basic distinction between multibyte and Unicode encodings.  Also
included are pointers to how XEmacs should make use of this API.

   The Win32 API is quite well-designed in its handling of strings
encoded for various character sets.  The API is geared around the idea
that two different methods of encoding strings should be supported.
These methods are called multibyte and Unicode, respectively.  The
multibyte encoding is compatible with ASCII strings and is a more
efficient representation when dealing with strings containing primarily
ASCII characters, but it has a great number of serious deficiencies and
limitations, including that it is very difficult and error-prone to work
with strings in this encoding, and any particular string in a multibyte
encoding can only contain characters from a very limited number of
character sets.  The Unicode encoding rectifies all of these
deficiencies, but it is not compatible with ASCII strings (in other
words, an existing program will not be able to handle the encoded
strings unless it is explicitly modified to do so), and it takes up
twice as much memory space as multibyte encodings when encoding a purely
ASCII string.

   Multibyte encodings use a variable number of bytes (either one or
two) to represent characters.  ASCII characters are also represented by
a single byte with its high bit not set, and non-ASCII characters are
represented by one or two bytes, the first of which always has its high
bit set.  (The second byte, when it exists, may or may not have its high
bit set.)  There is no single multibyte encoding.  Instead, there is
generally one encoding per non-ASCII character set.  Such an encoding is
capable of representing (besides ASCII characters, of course) only
characters from one (or possibly two) particular character sets.

   Multibyte encoding makes processing of strings very difficult.  For
example, given a pointer to the beginning of a character within a
string, finding the pointer to the beginning of the previous character
may require backing up all the way to the beginning of the string, and
then moving forward.  Also, an operation such as separating out the
components of a path by searching for backslashes will fail if it's
implemented in the simplest (but not multibyte-aware) fashion, because
it may find what appears to be a backslash, but which is actually the
second byte of a two-byte character.  Also, the limited number of
character sets that any particular multibyte encoding can represent
means that loss of data is likely if a string is converted from the
XEmacs internal format into a multibyte format.

   For these reasons, the C code in XEmacs should never do any sort of
work with multibyte encoded strings (or with strings in any external
encoding for that matter).  Strings should always be maintained in the
internal encoding, which is predictable, and converted to an external
encoding only at the point where the string moves from the XEmacs C
code and enters a system library function.  Similarly, when a string is
returned from a system library function, it should be immediately
converted into the internal coding before any operations are done on it.

   Unicode, unlike multibyte encodings, is a fixed-width encoding where
every character is represented using 16 bits.  It is also capable of
encoding all the characters from all the character sets in common use in
the world.  The predictability and completeness of the Unicode encoding
makes it a very good encoding for strings that may contain characters
from many character sets mixed up with each other.  At the same time, of
course, it is incompatible with routines that expect ASCII characters
and also incompatible with general string manipulation routines, which
will encounter a great number of what would appear to be embedded nulls
in the string.  It also takes twice as much room to encode strings
containing primarily ASCII characters.  This is why XEmacs does not use
Unicode or similar encoding internally for buffers.

   The Win32 API cleverly deals with the issue of 8 bit vs. 16 bit
characters by declaring a type called `"TCHAR"' which specifies a
generic character, either 8 bits or 16 bits.  Generally `TCHAR' is
defined to be the same as the simple C type `char', unless the
preprocessor constant `UNICODE' is defined, in which case `TCHAR' is
defined to be `WCHAR', which is a 16 bit type.  Nearly all functions in
the Win32 API that take strings are defined to take strings that are
actually arrays of `TCHAR's.  There is a type `LPTSTR' which is defined
to be a string of `TCHAR's and another type `LPCTSTR' which is a const
string of `TCHAR's.  The theory is that any program that uses `TCHAR's
exclusively to represent characters and does not make assumptions about
the size of a `TCHAR' or the way that the characters are encoded should
work transparently regardless of whether the `UNICODE' preprocessor
constant is defined, which is to say, regardless of whether 8 bit
multibyte or 16 bit Unicode characters are being used.  The way that
this is actually implemented is that every Win32 API function that takes
a string as an argument actually maps to one of two functions which are
suffixed with an `A' (which stands for ANSI, and means multibyte
strings) or `W' (which stands for wide, and means Unicode strings).
The mapping is, of course, controlled by the same `UNICODE'
preprocessor constant.  Generally all structures containing strings in
them actually map to one of two different kinds of structures, with
either an `A' or a `W' suffix after the structure name.

   Unfortunately, not all of the implementations of the Win32 API
implement all of the functionality described above.  In particular,
Windows 95 does not implement very much Unicode functionality.  It does
implement functions to convert multibyte-encoded strings to and from
Unicode strings, and provides Unicode versions of certain low-level
functions like `ExtTextOut()'.  In fact, all of the rest of the Unicode
versions of API functions are just stubs that return an error.
Conversely, all versions of Windows NT completely implement all the
Unicode functionality, but some versions (especially versions before
Windows NT 4.0) don't implement much of the multibyte functionality.
For this reason, as well as for general code cleanliness, XEmacs needs
to be written in such a way that it works with or without the `UNICODE'
preprocessor constant being defined.

   Getting XEmacs to run when all strings are Unicode primarily
involves removing any assumptions made about the size of characters.
Remember what I said earlier about how the point of conversion between
internally and externally encoded strings should occur at the point of
entry or exit into or out of a library function.  With this in mind, an
externally encoded string in XEmacs can be treated simply as an
arbitrary sequence of bytes of some length which has no particular
relationship to the length of the string in the internal encoding.

   #### The rest of this is *out-of-date* and needs to be written to
reference the actual coding systems or aliases that we currently use.

   [[ To facilitate this, the enum `external_data_format', which is
declared in `lisp.h', is expanded to contain three new formats, which
are `FORMAT_LOCALE', `FORMAT_UNICODE' and `FORMAT_TSTR'.
`FORMAT_LOCALE' always causes encoding into a multibyte string
consistent with the encoding of the current locale.  The functions to
handle locales are different under Unix and Windows and locales are a
process property under Unix and a thread property under Windows, but
the concepts are basically the same.  `FORMAT_UNICODE' of course causes
encoding into Unicode and `FORMAT_TSTR' logically maps to either
`FORMAT_LOCALE' or `FORMAT_UNICODE' depending on the `UNICODE'
preprocessor constant.

   Under Unix the behavior of `FORMAT_TSTR' is undefined and this
particular format should not be used.  Under Windows however
`FORMAT_TSTR' should be used for pretty much all of the Win32 API
calls.  The other two formats should only be used in particular APIs
that specifically call for a multibyte or Unicode encoded string
regardless of the `UNICODE' preprocessor constant.  String constants
that are to be passed directly to Win32 API functions, such as the
names of window classes, need to be bracketed in their definition with
a call to the macro `TEXT'.  This awfully named macro, which comes out
of the Win32 API, appropriately makes a string of either regular or
wide chars, which is to say this string may be prepended with an `L'
(causing it to be a wide string) depending on the `UNICODE'
preprocessor constant.

   By the way, if you're wondering what happened to `FORMAT_OS', I
think that this format should go away entirely because it is too vague
and should be replaced by more specific formats as they are defined.  ]]

   Use Qnative for Unix conversion, Qmswindows_tstr for Windows ...

   String constants that are to be passed directly to Win32 API
functions, such as the names of window classes, need to be bracketed in
their definition with a call to the macro XETEXT. This appropriately
makes a string of either regular or wide chars, which is to say this
string may be prepended with an L (causing it to be a wide string)
depending on XEUNICODE_P.


File: internals.info,  Node: Modules for Interfacing with MS Windows,  Next: CHANGES from 21.4-windows branch,  Prev: Windows I18N Introduction,  Up: Interface to MS Windows

41.4 Modules for Interfacing with MS Windows
============================================

There are two different general Windows-related include files in src.

   Uses are approximately:

`syswindows.h'
     Wrapper around `<windows.h>', including missing defines as
     necessary.  Includes stuff needed on both Cygwin and native
     Windows, regardless of window system chosen.  Includes definitions
     needed for Unicode conversion/encapsulation, and other
     Mule-related stuff, plus various other prototypes and
     Windows-specific, but not GUI-specific, stuff.

`console-msw.h'
     Used on both Cygwin and native Windows, but only when native window
     system (as opposed to X) chosen.  Includes `syswindows.h'.

   Summary of files:

`console-msw.h'
     include file for native windowing (otherwise, `console-x.h', etc.)

`console-msw.c, frame-msw.c, etc.'
     native windowing, as above

`process-nt.c'
     subprocess support for native OS (otherwise, `process-unix.c')

`nt.c'
     support routines used under native OS

`win32.c'
     support routines used under both OS environments

`syswindows.h'
     support header for both environments

`nt/xemacs.mak'
     Makefile for VC++ (otherwise, `src/Makefile.in.in')

`s/windowsnt.h'
     s header for basic native-OS defines, VC++ compiler

`s/mingw32.h'
     s header for basic native-OS defines, GCC/MinGW compiler

`s/cygwin.h'
     s header for basic Cygwin defines

`s/win32-native.h'
     s header for basic native-OS defines, all compilers

`s/win32-common.h'
     s header for defines for both OS environments

`intl-win32.c'
     internationalization functions for both OS environments

`intl-encap-win32.c'
     Unicode encapsulation functions for both OS environments

`intl-auto-encap-win32.c'
     Auto-generated Unicode encapsulation functions

`intl-auto-encap-win32.h'
     Auto-generated Unicode encapsulation headers


File: internals.info,  Node: CHANGES from 21.4-windows branch,  Prev: Modules for Interfacing with MS Windows,  Up: Interface to MS Windows

41.5 CHANGES from 21.4-windows branch (probably obsolete)
=========================================================

This node contains the `CHANGES-msw' log that Andy Piper kept while he
was maintaining the Windows branch of 21.4.  These changes have
(presumably) long since been merged to both 21.4 and 21.5, but let's not
throw the list away yet.

CHANGES-msw
===========

This file briefly describes all mswindows-specific changes to XEmacs in
the OXYMORON series of releases. The mswindows release branch contains
additional changes on top of the mainline XEmacs release. These changes
are deemed necessary for XEmacs to be fully functional under mswindows.
It is not intended that these changes cause problems on UNIX systems,
but they have not been tested on UNIX platforms. Caveat Emptor.

   See the file `CHANGES-release' for a full list of mainline changes.

to XEmacs 21.4.9 "Informed Management (Windows)"
================================================

   * Fix layout of widgets so that the search dialog works.

   * Fix focus capture of widgets under X.

to XEmacs 21.4.8 "Honest Recruiter (Windows)"
=============================================

   * All changes from 21.4.6 and 21.4.7.

   * Make sure revert temporaries are not visiting files. Suggested by
     Mike Alexander.

   * File renaming fix from Mathias Grimmberger.

   * Fix printer metrics on windows 95 from Jonathan Harris.

   * Fix layout of widgets so that the search dialog works.

   * Fix focus capture of widgets under X.

   * Buffers tab doc fixes from John Palmieri.

   * Sync with FSF custom `:set-after' behavior.

   * Virtual window manager freeze fix from Rick Rankin.

   * Fix various printing problems.

   * Enable windows printing on cygwin.

to XEmacs 21.4.7 "Economic Science (Windows)"
=============================================

   * All changes from 21.4.6.

   * Fix problems with auto-revert with noconfirm.

   * Undo autoconf 2.5x changes.

   * Undo 21.4.7 process change.

   to XEmacs 21.4.6 "Common Lisp (Windows)"

   * Made native registry entries match the installer.

   * Fixed mousewheel lockups.

   * Frame iconifcation fix from Adrian Aichner.

   * Fixed some printing problems.

   * Netinstaller updated to support kit revisions.

   * Fixed customize popup menus.

   * Fixed problems with too many dialog popups.

   * Netinstaller fixed to correctly upgrade shortcuts when upgrading
     core XEmacs.

   * Fix for virtual window managers from Adrian Aichner.

   * Installer registers all C++ file types.

   * Short-filename fix from Peter Arius.

   * Fix for GC assertions from Adrian Aichner.

   * Winclient DDE client from Alastair Houghton.

   * Fix event assert from Mike Alexander.

   * Warning removal noticed by Ben Wing.

   * Redisplay glyph height fix from Ben Wing.

   * Printer margin fix from Jonathan Harris.

   * Error dialog fix suggested by Thomas Vogler.

   * Fixed revert-buffer to not revert in the case that there is
     nothing to be done.

   * Glyph-baseline fix from Nix.

   * Fixed clipping of wide glyphs in non-zero-length extents.

   * Windows build fixes.

   * Fixed `:initial-focus' so that it works.

to XEmacs 21.4.5 "Civil Service (Windows)"
==========================================

   * Fixed a scrollbar problem when selecting the frame with focus.

   * Fixed `mswindows-shell-execute' under cygwin.

   * Added a new function `mswindows-cygwin-to-win32-path' for JDE.

   * Added support for dialog-based directory selection.

   * The installer version has been updated to the 21.5 netinstaller.
     The 21.5 installer now does proper dde file association and adds
     uninstall capability.

   * Handle leak fix from Mike Alexander.

   * New release build script.


File: internals.info,  Node: Interface to the X Window System,  Next: Dumping,  Prev: Interface to MS Windows,  Up: Top

42 Interface to the X Window System
***********************************

Mostly undocumented.

* Menu:

* Lucid Widget Library::        An interface to various widget sets.
* Modules for Interfacing with X Windows::


File: internals.info,  Node: Lucid Widget Library,  Next: Modules for Interfacing with X Windows,  Prev: Interface to the X Window System,  Up: Interface to the X Window System

42.1 Lucid Widget Library
=========================

Lwlib is extremely poorly documented and quite hairy.  The author(s)
blame that on X, Xt, and Motif, with some justice, but also sufficient
hypocrisy to avoid drawing the obvious conclusion about their own work.

   The Lucid Widget Library is composed of two more or less independent
pieces.  The first, as the name suggests, is a set of widgets.  These
widgets are intended to resemble and improve on widgets provided in the
Motif toolkit but not in the Athena widgets, including menubars and
scrollbars.  Recent additions by Andy Piper integrate some "modern"
widgets by Edward Falk, including checkboxes, radio buttons, progress
gauges, and index tab controls (aka notebooks).

   The second piece of the Lucid widget library is a generic interface
to several toolkits for X (including Xt, the Athena widget set, and
Motif, as well as the Lucid widgets themselves) so that core XEmacs
code need not know which widget set has been used to build the
graphical user interface.

* Menu:

* Generic Widget Interface::    The lwlib generic widget interface.
* Scrollbars::
* Menubars::
* Checkboxes and Radio Buttons::
* Progress Bars::
* Tab Controls::


File: internals.info,  Node: Generic Widget Interface,  Next: Scrollbars,  Prev: Lucid Widget Library,  Up: Lucid Widget Library

42.1.1 Generic Widget Interface
-------------------------------

In general in any toolkit a widget may be a composite object.  In Xt,
all widgets have an X window that they manage, but typically a complex
widget will have widget children, each of which manages a subwindow of
the parent widget's X window.  These children may themselves be
composite widgets.  Thus a widget is actually a tree or hierarchy of
widgets.

   For each toolkit widget, lwlib maintains a tree of `widget_values'
which mirror the hierarchical state of Xt widgets (including Motif,
Athena, 3D Athena, and Falk's widget sets).  Each `widget_value' has
`contents' member, which points to the head of a linked list of its
children.  The linked list of siblings is chained through the `next'
member of `widget_value'.

                +-----------+
                | composite |
                +-----------+
                      |
                      | contents
                      V
                  +-------+ next +-------+ next +-------+
                  | child |----->| child |----->| child |
                  +-------+      +-------+      +-------+
                                     |
                                     | contents
                                     V
                              +-------------+ next +-------------+
                              | grand child |----->| grand child |
                              +-------------+      +-------------+

     The `widget_value' hierarchy of a composite widget with two simple
     children and one composite child.

   The `widget_instance' structure maintains the inverse view of the
tree.  As for the `widget_value', siblings are chained through the
`next' member.  However, rather than naming children, the
`widget_instance' tree links to parents.

                +-----------+
                | composite |
                +-----------+
                      A
                      | parent
                      |
                  +-------+ next +-------+ next +-------+
                  | child |----->| child |----->| child |
                  +-------+      +-------+      +-------+
                                     A
                                     | parent
                                     |
                              +-------------+ next +-------------+
                              | grand child |----->| grand child |
                              +-------------+      +-------------+

     The `widget_value' hierarchy of a composite widget with two simple
     children and one composite child.

   This permits widgets derived from different toolkits to be updated
and manipulated generically by the lwlib library. For instance
`update_one_widget_instance' can cope with multiple types of widget and
multiple types of toolkit. Each element in the widget hierarchy is
updated from its corresponding `widget_value' by walking the
`widget_value' tree.  This has desirable properties.  For example,
`lw_modify_all_widgets' is called from `glyphs-x.c' and updates all the
properties of a widget without having to know what the widget is or
what toolkit it is from.  Unfortunately this also has its hairy
properties; the lwlib code quite complex. And of course lwlib has to
know at some level what the widget is and how to set its properties.

   The `widget_instance' structure also contains a pointer to the root
of its tree.  Widget instances are further confi


File: internals.info,  Node: Scrollbars,  Next: Menubars,  Prev: Generic Widget Interface,  Up: Lucid Widget Library

42.1.2 Scrollbars
-----------------


File: internals.info,  Node: Menubars,  Next: Checkboxes and Radio Buttons,  Prev: Scrollbars,  Up: Lucid Widget Library

42.1.3 Menubars
---------------


File: internals.info,  Node: Checkboxes and Radio Buttons,  Next: Progress Bars,  Prev: Menubars,  Up: Lucid Widget Library

42.1.4 Checkboxes and Radio Buttons
-----------------------------------


File: internals.info,  Node: Progress Bars,  Next: Tab Controls,  Prev: Checkboxes and Radio Buttons,  Up: Lucid Widget Library

42.1.5 Progress Bars
--------------------


File: internals.info,  Node: Tab Controls,  Prev: Progress Bars,  Up: Lucid Widget Library

42.1.6 Tab Controls
-------------------


File: internals.info,  Node: Modules for Interfacing with X Windows,  Prev: Lucid Widget Library,  Up: Interface to the X Window System

42.2 Modules for Interfacing with X Windows
===========================================

     Emacs.ad.h

   A file generated from `Emacs.ad', which contains XEmacs-supplied
fallback resources (so that XEmacs has pretty defaults).

     EmacsFrame.c
     EmacsFrame.h
     EmacsFrameP.h

   These modules implement an Xt widget class that encapsulates a frame.
This is for ease in integrating with Xt.  The EmacsFrame widget covers
the entire X window except for the menubar; the scrollbars are
positioned on top of the EmacsFrame widget.

   *Warning:* Abandon hope, all ye who enter here.  This code took an
ungodly amount of time to get right, and is likely to fall apart
mercilessly at the slightest change.  Such is life under Xt.

     EmacsManager.c
     EmacsManager.h
     EmacsManagerP.h

   These modules implement a simple Xt manager (i.e. composite) widget
class that simply lets its children set whatever geometry they want.
It's amazing that Xt doesn't provide this standardly, but on second
thought, it makes sense, considering how amazingly broken Xt is.

     EmacsShell-sub.c
     EmacsShell.c
     EmacsShell.h
     EmacsShellP.h

   These modules implement two Xt widget classes that are subclasses of
the TopLevelShell and TransientShell classes.  This is necessary to deal
with more brokenness that Xt has sadistically thrust onto the backs of
developers.

     xgccache.c
     xgccache.h

   These modules provide functions for maintenance and caching of GC's
(graphics contexts) under the X Window System.  This code is junky and
needs to be rewritten.

     select-msw.c
     select-x.c
     select.c
     select.h

   This module provides an interface to the X Window System's concept of
"selections", the standard way for X applications to communicate with
each other.

     xintrinsic.h
     xintrinsicp.h
     xmmanagerp.h
     xmprimitivep.h

   These header files are similar in spirit to the `sys*.h' files and
buffer against different implementations of Xt and Motif.

   * `xintrinsic.h' should be included in place of `<Intrinsic.h>'.

   * `xintrinsicp.h' should be included in place of `<IntrinsicP.h>'.

   * `xmmanagerp.h' should be included in place of `<XmManagerP.h>'.

   * `xmprimitivep.h' should be included in place of `<XmPrimitiveP.h>'.

     ExternalClient-Xlib.c
     ExternalClient.c
     ExternalClient.h
     ExternalClientP.h
     ExternalShell.c
     ExternalShell.h
     ExternalShellP.h
     extw-Xlib.c
     extw-Xlib.h
     extw-Xt.c
     extw-Xt.h

   These files provide the "external widget" interface, which allows an
XEmacs frame to appear as a widget in another application.  To do this,
you have to configure with `--external-widget'.

   `ExternalShell*' provides the server (XEmacs) side of the connection.

   `ExternalClient*' provides the client (other application) side of
the connection.  These files are not compiled into XEmacs but are
compiled into libraries that are then linked into your application.

   `extw-*' is common code that is used for both the client and server.

   Don't touch this code; something is liable to break if you do.


File: internals.info,  Node: Dumping,  Next: Future Work,  Prev: Interface to the X Window System,  Up: Top

43 Dumping
**********

* Menu:

* Dumping Justification::
* Overview::
* Data descriptions::
* Dumping phase::
* Reloading phase::


File: internals.info,  Node: Dumping Justification,  Next: Overview,  Prev: Dumping,  Up: Dumping

43.1 Dumping Justification
==========================

The C code of XEmacs is just a Lisp engine with a lot of built-in
primitives useful for writing an editor.  The editor itself is written
mostly in Lisp, and represents around 100K lines of code.  Loading and
executing the initialization of all this code takes a bit a time (five
to ten times the usual startup time of current xemacs) and requires
having all the lisp source files around.  Having to reload them each
time the editor is started would not be acceptable.

   The solution to this problem is called dumping: the build process
first creates the lisp engine under the name `temacs', then runs it
until it has finished loading and initializing all the lisp code, and
eventually creates a new executable called `xemacs' including both the
object code in `temacs', and, traditionally all the contents of the
memory after the initialization. This last feature was termed "unexec".

   Unexec, while it worked, had a huge problem: the creation of the new
executable from the actual contents of memory was an extremely
system-specific process, quite error-prone, and interfered with a lot
of system libraries (like malloc).  This got worse over time with
libraries using constructors which are automatically called when the
program is started (even before `main()'); those constructors tend to
crash when they were called multiple times, once before dumping and
once after (IRIX 6.x `libz.so' pulls in some C++ image libraries thru
dependencies which have this problem).

   Writing the dumper was also one of the most difficult parts of
porting XEmacs to a new operating system.

   Oliver Galibert replaced this approach by adding explicit
descrptions of objects in memory and implementing a "portable dumper",
which removes the need for unexec. See the rest of this section.


File: internals.info,  Node: Overview,  Next: Data descriptions,  Prev: Dumping Justification,  Up: Dumping

43.2 Overview
=============

The portable dumping system has to:

  1. At dump time, write all initialized, non-quickly-rebuildable data
     to a file [Note: currently named `xemacs.dmp', but the name will
     change], along with all information needed for the reloading.

  2. When starting xemacs, reload the dump file, relocate it to its new
     starting address if needed, and reinitialize all pointers to this
     data.  Also, rebuild all the quickly rebuildable data.

   Note: As of 21.5.18, the dump file has been moved inside of the
executable, although there are still problems with this on some systems.


File: internals.info,  Node: Data descriptions,  Next: Dumping phase,  Prev: Overview,  Up: Dumping

43.3 Data descriptions
======================

The more complex task of the dumper is to be able to write memory blocks
on the heap (lisp objects, i.e. lrecords, and C-allocated memory, such
as structs and arrays) to disk and reload them at a different address,
updating all the pointers they include in the process.  This is done by
using external data descriptions that give information about the layout
of the blocks in memory.

   The specification of these descriptions is in lrecord.h.  A
description of an lrecord is an array of struct memory_description.
Each of these structs include a type, an offset in the block and some
optional parameters depending on the type.  For instance, here is the
string description:

     static const struct memory_description string_description[] = {
       { XD_BYTECOUNT,         offsetof (Lisp_String, size) },
       { XD_OPAQUE_DATA_PTR,   offsetof (Lisp_String, data), XD_INDIRECT(0, 1) },
       { XD_LISP_OBJECT,       offsetof (Lisp_String, plist) },
       { XD_END }
     };

   The first line indicates a member of type Bytecount, which is used by
the next, indirect directive.  The second means "there is a pointer to
some opaque data in the field `data'".  The length of said data is
given by the expression `XD_INDIRECT(0, 1)', which means "the value in
the 0th line of the description (welcome to C) plus one".  The third
line means "there is a Lisp_Object member `plist' in the Lisp_String
structure".  `XD_END' then ends the description.

   This gives us all the information we need to move around what is
pointed to by a memory block (C or lrecord) and, by transitivity,
everything that it points to.  The only missing information for dumping
is the size of the block.  For lrecords, this is part of the
lrecord_implementation, so we don't need to duplicate it.  For C blocks
we use a struct sized_memory_description, which includes a size field
and a pointer to an associated array of memory_description.


File: internals.info,  Node: Dumping phase,  Next: Reloading phase,  Prev: Data descriptions,  Up: Dumping

43.4 Dumping phase
==================

Dumping is done by calling the function `pdump()' (in `dumper.c') which
is invoked from Fdump_emacs (in `emacs.c').  This function performs a
number of tasks.

* Menu:

* Object inventory::
* Address allocation::
* The header::
* Data dumping::
* Pointers dumping::


File: internals.info,  Node: Object inventory,  Next: Address allocation,  Prev: Dumping phase,  Up: Dumping phase

43.4.1 Object inventory
-----------------------

The first task is to build the list of the objects to dump.  This
includes:

   * lisp objects

   * other memory blocks (C structures, arrays. etc)

   We end up with one `pdump_block_list_elt' per object group (arrays
of C structs are kept together) which includes a pointer to the first
object of the group, the per-object size and the count of objects in the
group, along with some other information which is initialized later.

   These entries are linked together in `pdump_block_list' structures
and can be enumerated thru either:

  1. the `pdump_object_table', an array of `pdump_block_list', one per
     lrecord type, indexed by type number.

  2. the `pdump_opaque_data_list', used for the opaque data which does
     not include pointers, and hence does not need descriptions.

  3. the `pdump_desc_table', which is a vector of
     `memory_description'/`pdump_block_list' pairs, used for non-opaque
     C memory blocks.

   This uses a marking strategy similar to the garbage collector.  Some
differences though:

  1. We do not use the mark bit (which does not exist for generic
     memory blocks anyway); we use a big hash table instead.

  2. We use the external descriptions to follow pointers to generic
     memory blocks and opaque data in addition to Lisp_Object members,
     above and beyond what the garbage collector needs to do.

   This is done by `pdump_register_object()', which handles Lisp_Object
variables, and `pdump_register_block()' which handles generic memory
blocks (C structures, arrays, etc.), which both delegate the
description management to `pdump_register_sub()'. This latter function
also checks if the address of a Lisp_Object encountered is in
`staticpros', and if so, it moves the corresponding entry from
`staticpros' to `staticpros_nodump', since `staticpros' is relocated as
an array of pointers in the data segment and needs to be handled
differently to heap pointers.

   The hash table doubles as a map object to pdump_block_list_elmt (i.e.
allows us to look up a pdump_block_list_elmt with the object it points
to).  Entries are added with `pdump_add_block()' and looked up with
`pdump_get_block()'.  There is no need for entry removal.  The hash
value is computed quite simply from the object pointer by
`pdump_make_hash()'.

   The roots for the marking are:

  1. the `staticpro''ed variables (there is a special
     `staticpro_nodump()' call for protected variables we do not want to
     dump).

  2. the Lisp_Object variables registered via
     `dump_add_root_lisp_object' (`staticpro()' is equivalent to
     `staticpro_nodump()' + `dump_add_root_lisp_object()').

  3. the data-segment memory blocks registered via `dump_add_root_block'
     (for blocks with relocatable pointers), or `dump_add_opaque' (for
     "opaque" blocks with no relocatable pointers; this is just a
     shortcut for calling `dump_add_root_block' with a NULL
     description).

  4. the pointer variables registered via `dump_add_root_block_ptr',
     each of which points to a block of heap memory (generally a C
     structure or array).  Note that `dump_add_root_block_ptr' is not
     technically necessary, as a pointer variable can be seen as a
     special case of a data-segment memory block and registered using
     `dump_add_root_block'.  Doing it this way, however, would require
     another level of static structures declared.  Since pointer
     variables are quite common, `dump_add_root_block_ptr' is provided
     for convenience.  Note also that internally we have to treat it
     separately from `dump_add_root_block' rather than writing the
     former as a call to the latter, since we don't have support for
     creating and using memory descriptions on the fly - they must all
     be statically declared in the data-segment.

   This does not include the GCPRO'ed variables, the specbinds, the
catchtags, the backlist, the redisplay or the profiling info, since we
do not want to rebuild the actual chain of lisp calls which end up to
the dump-emacs call, only the global variables.

   Weak lists and weak hash tables are dumped as if they were their
non-weak equivalent (without changing their type, of course).  This has
not yet been a problem.


File: internals.info,  Node: Address allocation,  Next: The header,  Prev: Object inventory,  Up: Dumping phase

43.4.2 Address allocation
-------------------------

The next step is to allocate the offsets of each of the objects in the
final dump file.  This is done by `pdump_allocate_offset()' which is
called indirectly by `pdump_scan_by_alignment()'.

   The strategy to deal with alignment problems uses these facts:

  1. real world alignment requirements are powers of two.

  2. the C compiler is required to adjust the size of a struct so that
     you can have an array of them next to each other.  This means you
     can have an upper bound of the alignment requirements of a given
     structure by looking at which power of two its size is a multiple.

  3. the non-variant part of variable size lrecords has an alignment
     requirement of 4.

   Hence, for each lrecord type, C struct type or opaque data block the
alignment requirement is computed as a power of two, with a minimum of
2^2 for lrecords.  `pdump_scan_by_alignment()' then scans all the
`pdump_block_list_elmt''s, the ones with the highest requirements
first.  This ensures the best packing.

   The maximum alignment requirement we take into account is 2^8.

   `pdump_allocate_offset()' only has to do a linear allocation,
starting at offset 256 (this leaves room for the header and keeps the
alignments happy).


File: internals.info,  Node: The header,  Next: Data dumping,  Prev: Address allocation,  Up: Dumping phase

43.4.3 The header
-----------------

The next step creates the file and writes a header with a signature and
some random information in it.  The `reloc_address' field, which
indicates at which address the file should be loaded if we want to avoid
post-reload relocation, is set to 0.  It then seeks to offset 256 (base
offset for the objects).


File: internals.info,  Node: Data dumping,  Next: Pointers dumping,  Prev: The header,  Up: Dumping phase

43.4.4 Data dumping
-------------------

The data is dumped in the same order as the addresses were allocated by
`pdump_dump_data()', called from `pdump_scan_by_alignment()'.  This
function copies the data to a temporary buffer, relocates all pointers
in the object to the addresses allocated in step Address Allocation,
and writes it to the file.  Using the same order means that, if we are
careful with lrecords whose size is not a multiple of 4, we are ensured
that the object is always written at the offset in the file allocated
in step Address Allocation.


File: internals.info,  Node: Pointers dumping,  Prev: Data dumping,  Up: Dumping phase

43.4.5 Pointers dumping
-----------------------

A bunch of tables needed to reassign properly the global pointers are
then written.  They are:

  1. the pdump_root_block_ptrs dynarr

  2. the pdump_opaques dynarr

  3. a vector of all the offsets to the objects in the file that
     include a description (for faster relocation at reload time)

  4. the pdump_root_objects and pdump_weak_object_chains dynarrs.

   For each of the dynarrs we write both the pointer to the variables
and the relocated offset of the object they point to.  Since these
variables are global, the pointers are still valid when restarting the
program and are used to regenerate the global pointers.

   The `pdump_weak_object_chains' dynarr is a special case.  The
variables it points to are the head of weak linked lists of lisp objects
of the same type.  Not all objects of this list are dumped so the
relocated pointer we associate with them points to the first dumped
object of the list, or Qnil if none is available.  This is also the
reason why they are not used as roots for the purpose of object
enumeration.

   Some very important information like `lrecord_implementations_table'
is handled indirectly using `dump_add_opaque' or
`dump_add_root_block_ptr'.

   This is the end of the dumping part.


File: internals.info,  Node: Reloading phase,  Prev: Dumping phase,  Up: Dumping

43.5 Reloading phase
====================

43.5.1 File loading
-------------------

The dump file contains C function pointers and Lisp code and should be
regarded as executable C code. As a result, the preferred place for the
dump file is within the XEmacs executable, since this avoids security
worries at startup (see LD_LIBRARY_PATH and its setuid restrictions for
motivation; ld.so(8) on Linux); this also has the benefit that no path
searching is needed very early before we have any significant natural
language processing available.

   This easily achieved on MS Windows by storing it in a "resource" at
compile time and loading that at runtime.

   On other platforms this is more difficult. It is currently done using
"INCBIN", a reasonably portable assembler macro from Dale Weiler that
includes an external binary in the executable with a given alignment;
this requires re-linking the executable after the dump file is
generated. If the current compiler supports C23's `#embed' directive
that is used in preference to `INCBIN'.

   In these days of reproducible builds this works well. The previous
approach, from Olivier Galibert, binary-patched the generated
executable, which interacts poorly with signed executables, as is
standard with macOS and the ARM architecture.

   If DUMP_IN_EXEC is turned off, the file is found either in the same
directory in which the XEmacs executable was found (for in-place
execution) or in "exec-directory", usually under ../lib under the same
prefix. It is then mmap'ed in memory (which ensures a PAGESIZE
alignment, at least 4096), or if mmap is unavailable or fails, a
256-bytes aligned malloc is done and the file is loaded.

   Some variables are reinitialized from the values found in the header.

   The difference between the actual loading address and the
reloc_address is computed and will be used for relocating the Lisp
data. The difference between the address of Fcons at dump time and at
load time is computed and is used for relocating C function pointers,
and the difference between the address of lisp_object_description at
dump time and at load time is computed and used for relocating C data
pointers, necessary for ASLR support.

43.5.2 Putting back the pdump_opaques
-------------------------------------

The memory contents are restored in the obvious and trivial way.

43.5.3 Putting back the pdump_root_block_ptrs
---------------------------------------------

The variables pointed to by pdump_root_block_ptrs in the dump phase are
reset to the right relocated object addresses.

43.5.4 Object relocation
------------------------

All the objects are relocated using their description and their offset
by `pdump_reloc_one'.  This step is unnecessary if the reloc_address is
equal to the file loading address.

43.5.5 Putting back the pdump_root_objects and pdump_weak_object_chains
-----------------------------------------------------------------------

Same as Putting back the pdump_root_block_ptrs.

43.5.6 Reorganize the hash tables
---------------------------------

Since some of the hash values in the lisp hash tables are
address-dependent, their layout is now wrong.  So we go through each of
them and have them resorted by calling `pdump_reorganize_hash_table'.


File: internals.info,  Node: Future Work,  Next: Future Work Discussion,  Prev: Dumping,  Up: Top

44 Future Work
**************

* Menu:

* Future Work -- General Suggestions::
* Future Work -- Elisp Compatibility Package::
* Future Work -- Drag-n-Drop::
* Future Work -- Standard Interface for Enabling Extensions::
* Future Work -- Better Initialization File Scheme::
* Future Work -- Keyword Parameters::
* Future Work -- Property Interface Changes::
* Future Work -- Toolbars::
* Future Work -- Menu API Changes::
* Future Work -- Removal of Misc-User Event Type::
* Future Work -- Mouse Pointer::
* Future Work -- Version Number and Development Tree Organization::
* Future Work -- Improvements to the `xemacs.org' Website::
* Future Work -- Keybindings::
* Future Work -- Byte Code Snippets::
* Future Work -- Lisp Stream API::
* Future Work -- Multiple Values::
* Future Work -- Macros::
* Future Work -- Specifiers::
* Future Work -- Making Elisp Function Calls Faster::
* Future Work -- Lisp Engine Replacement::
* Future Work -- Better Rendering Support::


File: internals.info,  Node: Future Work -- General Suggestions,  Next: Future Work -- Elisp Compatibility Package,  Prev: Future Work,  Up: Future Work

44.1 Future Work - General Suggestions
======================================

Jamie Zawinski's XEmacs Wishlist
--------------------------------

This document is based on Jamie Zawinski's xemacs wishlist
(http://www.jwz.org/doc/xemacs-wishlist.html).        Throughout this
page, "I" refers to Jamie.

   The list has been substantially reformatted and edited to fit the
needs       of this site. If you have any soul at all, you'll go check
out the       original. OK? You should also check out some other
wishlists
(http://www.xemacs.org/Releases/Public-21.2/execution.html#wishlists).

About the List
..............

I've ranked these (roughly) from easiest to hardest; though of all of
them, I think the debugger improvements would be the most useful. I
think the combination of emacs+gdb is the best Unix development
environment currently available, but it's still lamentably primitive
and extremely frustrating (much like Unix itself), especially if you
know what kinds of features more modern integrated debuggers have.

XEmacs Wishlist
...............

*Improve the keyboard macro system.*
     Keyboard macros are one of the most useful concepts that emacs has
     to offer, but there's room for improvement.

    *Make it possible to embed one macro inside of another.*
          Often, I'll define a keyboard macro, and then realize that
          I've left something out, or that there's more that I need to
          do; for example, I may define a macro that does something to
          the current line, and then realize that I want to apply it to
          a lot of lines. So, I'd like this to work:

               C-x (
               ; start macro #1
               ...
               ; (do stuff)
               C-x )
               ; done with macro #1
               ...
               ; (do stuff)
               C-x (
               ; start macro #2
               C-x e
               ; execute macro #1 (splice it into macro #2)
               C-s foo
               ; move forward to the next spot
               C-x )
               ; done with macro #2
               C-u 1000 C-x e
               ; apply the new macro

          That is, simply, one should be able to wrap new text around an
          existing macro. I can't tell you how many times I've defined
          a complex macro but left out the "`C-n C-a'" at the end...

          Yes, you can accomplish this with M-x name-last-kbd-macro, but
          that's a pain. And it's also more permanent than I'd often
          like.

    *Make it possible to correct errors when defining a macro.*
          Right now, the act of defining a macro stops if you get an
          error while defining it, and all of the characters you've
          already typed into the macro are gone. It needn't be that
          way. I think that, when that first error occurs, the user
          should be given the option of taking the last command off of
          the macro and trying again.

          The macro-reader knows where the bounds of multi-character
          command sequences are, and it could even keep track of the
          corresponding undo records; rubbing out the previous entry on
          the macro could also undo any changes that command had made.
          (This should also work if the macro spans multiple buffers,
          and should restore window configurations as well.)

          You'd want multi-level undo for this as well, so maybe the
          way to go would be to add some new key sequence which was
          used only as the back-up-inside-a-keyboard-macro-definition
          command.

          I'm not totally sure that this would end up being very usable;
          maybe it would be too hard to deal with. Which brings us to:

    *Make it possible to edit a keyboard macro after it has been defined.*
          I only just discovered `edit-kbd-macro' (`C-x C-k').  It is
          very, very cool.

          The trick it does of showing the command which will be
          executed is somewhat error-prone, as it can only look up
          things in the current map or the global map; if the macro
          changed buffers, it wouldn't be displaying the right
          commands. (One of the things I often use macros for is
          operating on many files at once, by bringing up a dired buffer
          of those files, editing them, and then moving on to the next.)

          However, if the act of recording a macro also kept track of
          the actual commands that had gotten executed, it could make
          use of that info as well.

          Another way of editing a macro, other than as text in a
          buffer, would be to have a command which single-steps a
          macro: you would lean on the space bar to watch the macro
          execute one character (command?)  at a time, and then when
          you reached the point you wanted to change, you could do some
          gesture to either: insert some keystrokes into the middle of
          the macro and then continue; or to replace the rest of the
          macro from here to the end; or something.

          Another similar hack might be to convert a macro to the
          equivalent lisp code, so that one could tweak it later in
          ways that would be too hard to do from the keyboard (wrapping
          parts of it in `while' loops or something.) (`M-x
          insert-kbd-macro' isn't really what I'm talking about here: I
          mean insert the list of commands, not the list of keystrokes.)

*Save my wrists!*
     In the spirit of the ``teach-extended-commands-p'' variable, it
     would be interesting if emacs would keep track of what are the
     commands I use most often, perhaps grouped by proximity or mode -
     it would then be more obvious which commands were most likely
     candidates for placement on a toolbar, or popup menu, or just a
     more convenient key binding.

     Bonus points if it figures out that I type "`bt\n'" and
     "`ret\ny\n'" into my `*gdb*' buffer about a hundred thousand times
     a day.

*XmCreateFileSelectionBox*
     The thing that "File/Open..." pops up has excellent _hack_ value,
     but as a user interface, it's an abomination. Isn't it time
     someone added a real file selection dialog already? (For the
     Motifly-challenged, the Athena-based file selector that GhostView
     uses seems adequate.)

*Improve the toolbar system.*
     It's great that XEmacs has a toolbar, but it's damn near impossible
     to customize it.

    *Make it easy to define new toolbar buttons.*
          Currently, to define a toolbar button that has a text
          equivalent, one must edit a pixmap, and put the text there!
          That's prohibitive.  One should be able to add some kind of
          generic toolbar button, with a plain icon or none at all, but
          which has a text label, without having to use a paint program.

    *Make it easy to have customized, mode-local toolbars.*
          In my `c-mode-hook', for example, I can add a couple of new
          keybindings, and delete a few others, and to do that, I don't
          have to duplicate the entire definition of the `c-mode-map'.
          Making mode-local additions and subtractions to the toolbars
          should be as easy.

    *Make it easy to have customized, mode-local popup menus.*
          The same situation holds for the right-mouse-button popup
          menu; one should be able to add new commands to those menus
          without difficulty.  One problem is that each mode which does
          have a popup menu implements it in a different way...

*Make the External Widget work.*
     About half of the work is done to make a replacement for the
     `XmText' widget which offloads editing responsibility to an
     external Emacs process. Someone should finish that. The benefit
     here would be that then, any Motif program could be linked such
     that all editing happened with a real Emacs behind it. (If you're
     Athena-minded, flavor with `Text' instead of `XmText' - it's
     probably easy to make it work with both.)

     The part of this that is done already is the ability to run an
     Emacs screen on a Window object that has been created by another
     process (this is what the `ExternalClient.c' and `ExternalShell.c'
     stuff is.) What is left to be done is, adding the
     text-widget-editor aspects of this.

     First, the emacs screen being displayed on that window would have
     to be one without a modeline, and one which behaved sensibly in
     the context of "I am a small multi-line text area embedded in a
     dialog box" as opposed to "I am a full-on text editor and lord of
     all that I survey."

     Second, the API that the (non-emacs-aware) user of the `XmText'
     widget expects would need to be implemented: give the caller the
     ability to pull the edited text string back out, and so on.  The
     idea here being, hooking up emacs as the widget editor should be as
     transparent as possible.

*Bring the debugger interface into the eighties.*
     Some of you may have seen my `gdb-highlight.el' package, that I
     posted to gnu.emacs.sources last month. I think it's really cool,
     but there should be a lot more work in that direction.  For those
     of you who haven't seen it, what it does is watch text that gets
     inserted into the `*gdb*' buffer and make very nearly everything
     be clickable and have a context-sensitive menu. Generally, the
     types that are noticed are:

        * function names;

        * variable and parameter names;

        * structure slots;

        * source file names;

        * type names;

        * breakpoint numbers;

        * stack frame numbers.

     Any time one of those objects is presented in the `*gdb*' buffer,
     it is mousable. Clicking middle button on it takes some default
     action (edits the function, selects the stack frame, disables the
     breakpoint, ...) Clicking the right button pops up a menu of
     commands, including commands specific to the object under the
     mouse, and/or other objects on the same line.

     So that's all well and good, and I get far more joy out of what
     this code does for me than I expected, but there are still a bunch
     of limitations. The debugger interface needs to do much, much more.

    *Make gdbsrc-mode not suck.*
          The idea behind `gdbsrc-mode' is on the side of the angels:
          one should be able to focus on the source code and not on the
          debugger buffer, absolutely. But the implementation is just
          awful.

          First and foremost, it should not change "modes" (in the more
          general sense). Any commands that it defines should be on
          keys which are exclusively used for that purpose, not keys
          which are normally self-inserting. I can't be the only person
          who usually has occasion to actually _edit_ the sources which
          the debugger has chosen to display! Switching into and out of
          `gdbsrc-mode' is prohibitive.

          I want to be looking at my sources at all times, yet I don't
          want to have to give up my source-editing gestures. I think
          the right way to accomplish this is to put the gdbsrc
          commands on the toolbar and on popup menus; or to let the
          user define their own keys (I could see devoting my
          <kp_enter> key to "step", or something common like that.)

          Also it's extremely frustrating that one can't turn off
          gdbsrc mode once it has been loaded, without exiting and
          restarting emacs; that alone means that I'd probably never
          take the time to learn how to use it, without first having
          taken the time to repair it...

    *Make it easier access to variable values.*
          I want to be able to double-click on a variable name to
          highlight it, and then drag it to the debugger window to have
          its value printed.

          I want gestures that let me write as well as read: for
          example, to store value A into slot B.

    *Make all breakpoints visible.*
          Any time there is a running gdb which has breakpoints, the
          buffers holding the lines on which those breakpoints are set
          should have icons in them. These icons should be
          context-sensitive: I should be able to pop up a menu to
          enable or disable them, to delete them, to change their
          commands or conditions.

          I should also be able to _move_ them. It's annoying when you
          have a breakpoint with a complex condition or command on it,
          and then you realize that you really want it to be at a
          different location. I want to be able to drag-and-drop the
          icon to its new home.

    *Make a debugger status display window.*
             *  I want a window off to the side that shows persistent
               information - it should have a pane which is a
               drag-editable, drag-reorderable representation of the
               elements on gdb's "display" list; they should be
               displayed here instead of being just dumped in with the
               rest of the output in the `*gdb*' buffer.

             *  I want a pane that displays the current call-stack and
               nothing else. I want a pane that displays the arguments
               and locals of the currently-selected frame and nothing
               else. I want these both to update as I move around on
               the stack.

             *  Since the unfortunate reality is that excavating this
               information from gdb can be slow, it would be a good
               idea for these panes to have a toggle button on them
               which meant "stop updating", so that when I want to move
               fast, I can, but I can easily get the display back when
               I need it again.

          The reason for all of this is that I spend entirely too much
          time scrolling around in the `*gdb*' buffer; with
          gdb-highlight, I can just click on a line in the backtrace
          output to go to that frame, but I find that I spend a lot of
          time _looking_ for that backtrace: since it's mixed in with
          all the other random output, I waste time looking around for
          things (and usually just give up and type "`bt'" again, then
          thrash around as the buffer scrolls, and I try to find the
          lower frames that I'm interested in, as they have invariably
          scrolled off the window already...

    *Save and restore breakpoints across emacs/debugger sessions.*
          This would be especially handy given that gdb leaks like a
          sieve, and with a big program, I only get a few dozen
          relink-and-rerun attempts before gdb has blown my swap space.

    *Keep breakpoints in sync with source lines.*
          When a program is recompiled and then reloaded into gdb, the
          breakpoints often end up in less-than-useful places. For
          example, when I edit text which occurs in a file anywhere
          before a breakpoint, emacs is aware that the line of the bp
          hasn't changed, but just that it is in a different place
          relative to the top of the file. Gdb doesn't know this, so
          your breakpoints end up getting set in the wrong places
          (usually the maximally inconvenient places, like _after_ a
          loop instead of _inside_ it). But emacs knows, so emacs should
          inform the debugger, and move the breakpoints back to the
          places they were intended to be.

     (Possibly the OOBR stuff does some of this, but can't tell, because
     I've never been able to get it to do anything but beep at me and
     mumble about environments. I find it pretty funny that the manual
     keeps explaining to me how intuitive it is, without actually
     giving me a clue how to launch it...)

*Add better dialog box features.*
     It'd be nice to be able to create more complex dialog boxes from
     emacs-lisp: ones with checkboxes, radio button groups, text
     fields, and popup menus.

*Add embeddable dialog boxes.*
     One of the things that the now-defunct Energize code (the C side of
     it, that is) could do was embed a dialog box between the toolbar
     and the main text area - buffers could have control panels
     associated with them, that had all kinds of complex behavior.

*Make the mark-stack be visible.*
     You know, I've encountered people who have been using emacs for
     years, and never use the mark stack for navigation. I can't live
     without it; "`C-u C-SPC'" is among my most common gestures.

       1.  It would be a lot easier to realize what's going to happen
          if the marks on the mark stack were visible. They could be
          displayed as small "caret" glyphs, for example; something
          large enough to be visible, but not easily mistaken for a
          character or for the cursor.

       2.  The marks and the selected region should be visible in the
          scrollbar as well - I don't remember where I first saw this
          idea, but it's very cool: there's a second,
          less-strongly-rendered "thumb" in the scrollbar which
          indicates the position and size of the selection; and there
          are tiny tick-marks which indicate the positions of the saved
          points.

       3.  Markers which are in registers (`point-to-register', `C-x
          /') should be displayed differently (more prominent.)

       4.  It'd be cool if you could pick up markers and move them
          around, to adjust the points you'll be coming back to later.

*Write a new garbage collector.*
     The emacs GC is very primitive; it is also, fortunately, a rather
     well isolated module, and it would not be a very big task to swap
     it with a new one (once that new one was written, that is.) Someone
     should go bone up on modern GC techniques, and then just dive right
     in...

*Add support for lexical scope to the emacs-lisp runtime.*
     Yadda yadda, this list goes to eleven.

Subject: *Re: XEmacs wishlist* Date: Wed, 14 May 1997 16:18:23 -0700
From: Jamie Zawinski <jwz@netscape.com> Newsgroups: comp.emacs.xemacs,
comp.emacs

   Andreas Schwab wrote:

     _Use `C-u C-x (': _

     _start-kbd-macro:
     Non-nil arg (prefix arg) means append to last macro defined; This
     begins by re-executing that macro as if you typed it again. _

   Cool, I didn't know it did that...

   But it only lets you append. I often want to prepend, or embed the
macro multiple times (motion 1, C-x e, motion 2, C-x e, motion 3.)

21.2 Showstoppers
-----------------

Author: Ben Wing (mailto:ben@xemacs.org)

   DISTRIBUTION ISSUES

   A. Unified Source Tarball.

   Packages go under root/lib/xemacs/xemacs-packages and no one ever has
to mess with -package-path and the result can be moved from one
directory to another pre- or post-install.

   Unified Binary Tarballs with Packages.

   Same principles as above.

   If people complain, we can also provide split binary tarballs
(architecture dependent and independent) and place these files in a
subdirectory so as not to confuse the majority just looking for one
tarball.

   Under Windows, we need to provide a WISE-style GUI setup program.
It's already there but needs some work so you can select "all" packages
easily (should be the default).

   Parallel Root and Package Trees.

   If the user downloads separately, the main source and the packages,
he will naturally untar them into the same directory. This results in
the parallel root and package structure. We should support this as a
"last resort," i.e., if we find no packages anywhere and are about to
resign ourselves to not having packages, then look for a parallel
package tree. The user who sets things up like this should be able to
either run in place or "make install" and get a proper installed
XEmacs. Never should the user have to touch -package-path.

   II. WINDOWS PRINTING

   Looks like the internals are done but not the GUI. This must be
working in 21.2.

   III. WINDOWS MULE

   Basic support should be there. There's already a patch to get things
started and I'll be doing more work to make this real.

   IV. GUTTER ETC.

   This stuff needs to be "stable" and generally free from bugs. Any
APIs we create need to be well-reviewed or marked clearly as
experimental.

   V. PORTABLE DUMPER

   Last bits need to be cleaned up. This should be made the "default"
for a while to flush-out problems. Under Microsoft Windows, Portable
Dumper must be the default in 21.2 because of the problems with the
existing dump process.

   COMMENT: I'd like to feature freeze this pretty soon and create a
21.3 tree where all of my major overhauls of Mule-related stuff will go
in. At the same time or around, we need to do the move-around in the
repository (or create a new one) and "upgrade" to the latest CVS server.


File: internals.info,  Node: Future Work -- Elisp Compatibility Package,  Next: Future Work -- Drag-n-Drop,  Prev: Future Work -- General Suggestions,  Up: Future Work

44.2 Future Work - Elisp Compatibility Package
==============================================

Author: Ben Wing (mailto:ben@xemacs.org)

   A while ago I created a package called Sysdep, which aimed to be a
forward compatibility package for Elisp.  The idea was that instead of
having to write your package using the oldest version of Emacs that you
wanted to support, you could use the newest XEmacs API, and then simply
load the Sysdep package, which would automatically define the new API in
terms of older APIs as necessary.  The idea of this package was good,
but its design wasn't perfect, and it wasn't widely adopted.  I propose
a new package called Compat that corrects the design flaws in Sysdep,
and hopefully will be adopted by most of the major packages.

   In addition, this package will provide macros that can be used to
bracket code as necessary to disable byte compiler warnings generated as
a result of supporting the APIs of different versions of Emacs; or
rather the Compat package strives to provide useful constructs to make
doing this support easier, and these constructs have the side effect of
not causing spurious byte compiler warnings.  The idea here is that it
should be possible to create well-written, clean, and understandable
Elisp that supports both older and newer APIs, and has no byte compiler
warnings.  Currently many warnings are unavoidable, and as a result,
they are simply ignored, which also causes a lot of legitimate warnings
to be ignored.

   The approach taken by the Sysdep package to make sure that the newest
API was always supported was fairly simple: when the Sysdep package was
loaded, it checked for the existence of new API functions, and if they
weren't defined, it defined them in terms of older API functions that
were defined.  This had the advantage that the checks for which API
functions were defined were done only once at load time rather than each
time the function was called.  However, the fact that the new APIs were
globally defined caused a lot of problems with unwanted interactions,
both with other versions of the Sysdep package provided as part of other
packages, and simply with compatibility code of other sorts in packages
that would determine whether an API existed by checking for the
existence of certain functions within that API.  In addition, the Sysdep
package did not scale well because it defined all of the functions that
it supported, regardless of whether or not they were used.

   The Compat package remedies the first problem by ensuring that the
new APIs are defined only within the lexical scope of the packages that
actually make use of the Compat package.  It remedies the second problem
by ensuring that only definitions of functions that are actually used
are loaded.  This all works roughly according to the following scheme:

  1.  Part of the Compat package is a module called the Compat
     generator.  This module is actually run as an additional step
     during byte compilation of a package that uses Compat.  This can
     happen either through the makefile or through the use of an
     `eval-when-compile' call within the package code itself.  What the
     generator does is scan all of the Lisp code in the package,
     determine which function calls are made that the Compat package
     knows about, and generates custom `compat' code that conditionally
     defines just these functions when the package is loaded.  The
     custom `compat' code can either be written to a separate Lisp file
     (for use with multi-file packages), or inserted into the beginning
     of the Lisp file of a single file package.  (In the latter case,
     the package indicates where this generated code should go through
     the use of magic comments that mark the beginning and end of the
     section.  Some will say that doing this trick is bad juju, but I
     have done this sort of thing before, and it works very well in
     practice).

  2.  The functions in the custom `compat' code have their names
     prefixed with both the name of the package and the word `compat',
     ensuring that there will be no name space conflicts with other
     functions in the same package, or with other packages that make
     use of the Compat package.

  3.  The actual definitions of the functions in the custom `compat'
     code are determined at run time.  When the equivalent API already
     exists, the wrapper functions are simply defined directly in terms
     of the actual functions, so that the only run time overhead from
     using the Compat package is one additional function call.
     (Alternatively, even this small overhead could be avoided by
     retrieving the definitions of the actual functions and supplying
     them as the definitions of the wrapper functions.  However, this
     appears to me to not be completely safe.  For example, it might
     have bad interactions with the advice package).

  4.  The code that wants to make use of the custom `compat' code is
     bracketed by a call to the construct `compat-execute'.  What this
     actually does is lexically bind all of the function names that are
     being redefined with macro functions by using the Common Lisp
     macro macrolet.  (The definition of this macro is in the CL
     package, but in order for things to work on all platforms, the
     definition of this macro will presumably have to be copied and
     inserted into the custom `compat' code).


   In addition, the Compat package should define the macro
`compat-if-fboundp'.  Similar macros such as `compile-when-fboundp' and
`compile-case-fboundp' could be defined using similar principles).  The
`compat-if-fboundp' macro behaves just like an `(if (fboundp ...) ...)'
clause when executed, but in addition, when it's compiled, it ensures
that the code inside the `if-true' sub-block will not cause any byte
compiler warnings about the function in question being unbound.  I
think that the way to implement this would be to make
`compat-if-fboundp' be a macro that does what it's supposed to do, but
which defines its own byte code handler, which ensures that the
particular warning in question will be suppressed.  (Actually ensuring
that just the warning in question is suppressed, and not any others,
might be rather tricky.  It certainly requires further thought).

   Note: An alternative way of avoiding both warnings about unbound
functions and warnings about obsolete functions is to just call the
function in question by using `funcall', instead of calling the
function directly.  This seems rather inelegant to me, though, and
doesn't make it obvious why the function is being called in such a
roundabout manner.  Perhaps the Compat package should also provide a
macro `compat-funcall', which works exactly like `funcall', but which
indicates to anyone reading the code why the code is expressed in such
a fashion.

   If you're wondering how to implement the part of the Compat generator
where it scans Lisp code to find function calls for functions that it
wants to do something about, I think the best way is to simply process
the code using the Lisp function `read' and recursively descend any
lists looking for function names as the first element of any list
encountered.  This might extract out a few more functions than are
actually called, but it is almost certainly safer than doing anything
trickier like byte compiling the code, and attempting to look for
function calls in the result.  (It could also be argued that the names
of the functions should be extracted, not only from the first element of
lists, but anywhere `symbol' occurs.  For example, to catch places
where a function is called using `funcall' or `apply'.  However, such
uses of functions would not be affected by the surrounding macrolet
call, and so there doesn't appear to be any point in extracting them).


File: internals.info,  Node: Future Work -- Drag-n-Drop,  Next: Future Work -- Standard Interface for Enabling Extensions,  Prev: Future Work -- Elisp Compatibility Package,  Up: Future Work

44.3 Future Work - Drag-n-Drop
==============================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* I propose completely redoing the drag-n-drop interface
to make it powerful and extensible enough to support such concepts as
drag over and drag under visuals and context menus invoked when a drag
is done with the right mouse button, to allow drop handlers to be
defined for all sorts of graphical elements including buffers, extents,
mode lines, toolbar items, menubar items, glyphs, etc., and to allow
different packages to add and remove drop handlers for the same drop
sites without interfering with each other.  The changes are extensive
enough that I think they can only be implemented in version 22, and the
drag-n-drop interface should remain experimental until then.

   The new drag-n-drop interface centers around the twin concepts of
"drop site" and "drop handler".  A "drop site" specifies a particular
graphical element where an object can be dropped onto, and a "drop
handler" encapsulates all of the behavior that happens when such an
object is dragged over and dropped onto a drop site.

   Each drop site has an object associated with it which is passed to
functions that are part of the drop handlers associated with that site.
The type of this object depends on the graphical element that comprises
the drop site.  The drop site object can be a buffer, an extent, a
glyph, a menu path, a toolbar item path, etc.  (These last two object
types are defined in Lisp Interface Changes (lisp-interface.html) in
the sections on menu and toolbar API changes.  If we wanted to allow
drops onto other kinds of drop sites, for example mode lines, we would
have to create corresponding path objects).  Each such object type
should be able to be accessed using the generalized property interface
defined above, and should have a property called `drop-handlers'
associated with it that specifies all of the drop handlers associated
with the drop site.  Normally, this property is not accessed directly,
but instead by using the drop handler API defined below, and Lisp
packages should not make any assumptions about the format of the data
contained in the `drop-handlers' property.

   Each drop handler has an object of type `drop-handler' associated
with it, whose primary purpose is to be a container for the various
properties associated with a particular drop handler.  These could
include, for example, a function invoked when the drop occurs, a context
menu invoked when a drop occurs as a result of a drag with the right
mouse button, functions invoked when a dragged object enters, leaves, or
moves within a drop site, the shape that the mouse pointer changes to
when an object is dragged over a drop site that allows this particular
object to be dropped onto it, the MIME types (actually a regular
expression matching the MIME types) of the allowable objects that can be
dropped onto the drop site, a "package tag" (a symbol specifying the
package that created the drop handler, used for identification
purposes), etc.  The drop handler object is passed to the functions that
are invoked as a result of a drag or a drop, most likely indirectly as
one of the properties of the drag or drop event passed to the function.
Properties of a drop handler object are accessed and modified in the
standard fashion using the generalized property interface.

   A drop handler is added to a drop site using the `add-drop-handler'
function.  The drop handler itself can either be created separately
using the `make-drop-handler' function and then passed in as one of the
parameters to `add-drop-handler', or it will be created automatically
by the `add-drop-handler' function, if the drop handler argument is
omitted, but keyword arguments corresponding to the valid keyword
properties for a drop handler are specified in the `add-drop-handler'
call.  Other functions, such as `find-drop-handler', `add-drop-handler'
(when specifying a drop handler before which the drop handler in
question is to be added), `remove-drop-handler' etc. should be defined
with obvious semantics.  All of these functions take or return a drop
site object which, as mentioned above, can be one of several object
types corresponding to graphical elements.  Defined drop handler
functions locate a particular drop handler using either the `MIME-type'
or `package-tag' property of the drop handler, as defined above.

   Logically, the drop handlers associated with a particular drop site
are an ordered list.  The first drop handler whose specified MIME type
matches the MIME type of the object being dragged or dropped controls
what happens to this object.  This is important particularly because the
specified MIME type of the drop handler can be a regular expression
that, for example, matches all audio objects with any sub-type.

   In the current drag-n-drop API, there is a distinction made between
objects with an associated MIME type and objects with an associated URL.
I think that this distinction is arbitrary, and should not exist.  All
objects should have a MIME type associated with them, and a new
XEmacs-specific MIME type should be defined for URLs, file names, etc.
as necessary.  I am not even sure that this is necessary, however, as
the MIME specification may specify a general concept of a pointer or
link to an object, which is exactly what we want.  Also in some cases
(for example, the name of a file that is locally available), the pointer
or link will have another MIME type associated with it, which is the
type of the object that is being pointed to.  I am not quite sure how we
should handle URL and file name objects being dragged, but I am positive
that it needs to be integrated with the mechanism used when an object
itself is being dragged or dropped.

   As is described in a separate page (misc-user-event.html), the
`misc-user-event' event type should be removed and split up into a
number of separate event types.  Two such event types would be
`drag-event' and `drop-event'.  A drop event is used when an object is
actually dropped, and a drag event is used if a function is invoked as
part of the dragging process.  (Such a function would typically be used
to control what are called "drag under visuals", which are changes to
the appearance of the drop site reflecting the fact that a compatible
object is being dragged over it).  The drag events and drop events
encapsulate all of the information that is pertinent to the drag or
drop action occurring, including such information as the actual MIME
type of the object in question, the drop handler that caused a function
to be invoked, the mouse event (or possibly even a keyboard event)
corresponding to the user's action that is causing the drag or drop,
etc.  This event is always passed to any function that is invoked as a
result of the drag or drop.  There should never be any need to refer to
the `current-mouse-event' variable, and in fact, this variable should
not be changed at all during a drag or a drop.


File: internals.info,  Node: Future Work -- Standard Interface for Enabling Extensions,  Next: Future Work -- Better Initialization File Scheme,  Prev: Future Work -- Drag-n-Drop,  Up: Future Work

44.4 Future Work - Standard Interface for Enabling Extensions
=============================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* Apparently, if you know the name of a package (for
example, `fusion'), you can load it using the `require' function, but
there's no standard way to turn it on or turn it off.  The only way to
figure out how to do that is to go read the source file, where
hopefully the comments at the start tell you the appropriate magic
incantations that you need to run in order to turn the extension on or
off.  There really needs to be standard functions, such as
`enable-extension' and `disable-extension', to do this sort of thing.
It seems like a glaring omission that this isn't currently present, and
it's really surprising to me that nobody has remarked on this.

   The easy part of this is defining the interface, and I think it
should be done as soon as possible.  When the package is loaded, it
simply calls some standard function in the package system, and passes
it the names of enable and disable functions, or perhaps just one
function that takes an argument specifying whether to enable or
disable.  In any case, this data is kept in a table which is used by the
`enable-extension' and `disable-extension' function.  There should also
be functions such as `extension-enabled-p' and
`enabled-extension-list', and so on with obvious semantics.  The hard
part is actually getting packages to obey this standard interface, but
this is mitigated by the fact that the changes needed to support this
interface are so simple.

   I have been conceiving of these enabling and disabling functions as
turning the feature on or off globally.  It's probably also useful to
have a standard interface returning a extension on or off in just the
particular buffer.  Perhaps then the appropriate interface would involve
registering a single function that takes an argument that specifies
various things, such as turn off globally, turn on globally, turn on or
off in the current buffer, etc.

   Part of this interface should specify the correct way to define
global key bindings.  The correct rule for this, of course, is that the
key bindings should not happen when the package is loaded, which is
often how things are currently done, but only when the extension is
actually enabled.  The key bindings should go away when the extension is
disabled.  I think that in order to support this properly, we should
expand the keymap interface slightly, so that in addition to other
properties associated with each key binding is a list of shadow
bindings.  Then there should be a function called
`define-key-shadowing', which is just like `define-key' but which also
remembers the previous key binding in a shadow list.  Then there can be
another function, something like `undefine-key', which restores the
binding to the most recently added item on the shadow list.  There are
already hash tables associated with each key binding, and it should be
easy to stuff additional values, such as a shadow list, into the hash
table.  Probably there should also be functions called
`global-set-key-shadowing' and `global-unset-key-shadowing' with
obvious semantics.

   Once this interface is defined, it should be easy to expand the
custom package so it knows about this interface.  Then it will be
possible to put all sorts of extensions on the options menu so that
they could be turned off and turned on very easily, and then when you
save the options out to a file, the design settings for whether these
extensions are enabled or not are saved out with it.  A whole lot of
custom junk that's been added to a lot of different packages could be
removed.  After doing this, we might want to think of a way to classify
extensions according to how likely we think the user will want to use
them.  This way we can avoid the problem of having a list of 100
extensions and the user not being able to figure out which ones might
be useful.  Perhaps the most useful extensions would appear immediately
on the extensions menu, and the less useful ones would appear in a
submenu of that, and another submenu might contain even less useful
extensions.  Of course the package authors might not be too happy with
this, but the users probably will be.  I think this at least deserves a
thought, although it's possible you might simply want to maintain a
list on the web site of extensions and a judgment on first of all, how
commonly a user might want this extension, and second of all, how well
written and bug-free the package is.  Both of these sorts of judgments
could be obtained by doing user surveys if need be.


File: internals.info,  Node: Future Work -- Better Initialization File Scheme,  Next: Future Work -- Keyword Parameters,  Prev: Future Work -- Standard Interface for Enabling Extensions,  Up: Future Work

44.5 Future Work - Better Initialization File Scheme
====================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* A proposal is outlined for converting XEmacs to use the
`.xemacs' subdirectory for its initialization files instead of putting
them in the user's home directory.  In the process, a general
pre-initialization scheme is created whereby all of the initialization
parameters, such as the location of the initialization files, whether
these files are loaded or not, where the initial frame is created, etc.
that are currently specified by command line arguments, by environment
variables, and other means, can be specified in a uniform way using
Lisp code.  Reasonable default behavior for everything will still be
provided, and the older, simpler means can be used if desired.
Compatibility with the current location and name of the initialization
file, and the current ill-chosen use for the `.xemacs' directory is
maintained, and the problem of how to gracefully migrate a user from the
old scheme into the new scheme while still allowing the user to use GNU
Emacs or older versions of XEmacs is solved.  A proposal for changing
the way that the initial frame is mapped is also outlined; this would
allow the user's initialization file to control the way that the initial
frame appears without resorting to hacks, while still making echo area
messages visible as they appear, and allowing the user to debug errors
in the initialization file.

Principles in the new scheme
----------------------------

  1.  XEmacs has a defined "pre-initialization process".  This process,
     whose purpose is to compute the values of the parameters that
     control how the initializiaton process proceeds, occurs as early
     as possible after the Lisp engine has been initialized, and in
     particular, it occurs before any devices have been opened, or
     before any initialization parameters are set that could reasonably
     be expected to be changed.  In fact, the pre-initialization
     process should take care of setting these parameters.  The code
     that implements the pre-initialization process should be written
     in Lisp and should be called from the Lisp function
     `normal-top-level', and the general way that the user customizes
     this process should also be done using Lisp code.

  2.  The pre-initialization process involves a number of properties,
     for example the directory containing the user initialization files
     (normally the `.xemacs' subdirectory), the name of the user init
     file, the name of the custom init file, where and what type the
     initial device is, whether and when the initial frame is mapped,
     etc.  A standard interface is provided for getting and setting the
     values of these properties using functions such as
     `set-pre-init-property', `pre-init-property', etc.  At various
     points during the pre-initialization process, the value of many of
     these properties can be undecided, which means that at the end of
     the process, the value of these properties will be derived from
     other properties in some fashion that is specific to each property.

  3.  The default values of these properties are set first from the
     registry under Windows, then from environment variables, then from
     command line switches, such as `-q' and `-nw'.

  4.  One of the command line switches is `-pre-init', whose value is a
     Lisp expression to be evaluated at pre-initialization time,
     similar to the `-eval' command line switch.  This allows any
     pre-initialization property to be set from the command line.

  5.  Let's define the term "to determine a pre-initialization
     property" to mean if the value of a property is undetermined, it
     is computed and set according to a rule that is specific to the
     property.  Then after the pre-init properties are initialized from
     the registry, from the environment variables, from command line
     arguments, two of the pre-init properties (specifically the init
     file directory and the location of the "pre-init file") are
     determined.  The purpose of the pre-init file is to contain Lisp
     code that is run at pre-initialization time, and to control how
     the initialization proceeds.  It is a bit similar to the standard
     init file, but the code in the pre-init file shouldn't do anything
     other than set pre-init properties.  Executing any code that does
     I/O might not produce expected results because the only device that
     will exist at the time is probably a stream device connected to the
     standard I/O of the XEmacs process.

  6.  After the pre-init file has been run, all of the rest of the
     pre-init properties are determined, and these values are then used
     to control the initialization process.  Some of the rules used in
     determining specific properties are:

       1.  If the `.xemacs' sub-directory exists, and it's not
          obviously a package root (which probably means that it
          contains a file like `init.el' or `pre-init.el', or if
          neither of those files is present, then it doesn't contain
          any sub-directories or files that look like what would be in
          a package root), then it becomes the value of the init file
          directory.  Otherwise the user's home directory is used.

       2.  If the init file directory is the user's home directory,
          then the init file is called `.emacs'.  Otherwise, it's
          called `init.el'.

       3.  If the init file directory is the user's home directory,
          then the pre-init file is called `.xemacs-pre-init.el'.
          Otherwise it's called `pre-init.el'. (One of the reasons for
          this rule has to do with the dialog box that might be
          displayed at startup.  This will be described below.)

       4.  If the init file directory is the user's home directory,
          then the custom init file is called `.xemacs-custom-init.el'.
          Otherwise, it's called `custom-init.el'.


  7.  After the first normal device is created, but before any frames
     are created on it, the XEmacs initialization code checks to see if
     the old init file scheme is being used, which is to say that the
     init file directory is the same as the user's home directory.  If
     that's the case, then normally a dialog box comes up (or a
     question is asked on the terminal if XEmacs is being run in a
     non-windowing mode) which asks if the user wants to migrate his
     initialization files to the new scheme.  The possible responses
     are *Yes*, *No*, and *No, and don't ask this again*.  If this last
     response is chosen, then the file `.xemacs-pre-init.el' in the
     user's home directory is created or appended to with a line of
     Lisp code that sets up a pre-init property indicating that this
     dialog box shouldn't come up again.  If the *Yes* option is
     chosen, then any package root files in `.xemacs' are moved into
     `.xemacs/packages', the file `.emacs' is moved into
     `.xemacs/init.el' and `.emacs' in the home directory becomes a
     symlink to this file.  This way some compatibility is still
     maintained with GNU Emacs and older versions of XEmacs.  The code
     that implements this has to be written very carefully to make sure
     that it doesn't accidentally delete or mess up any of the files
     that get moved around.


The custom init file
--------------------

The "custom init file" is where the custom package writes its options.
This obviously needs to be a separate file from the standard init file.
It should also be loaded before the init file rather than after, as is
usually done currently, so that the init file can override these
options if it wants to.

Frame mapping
-------------

In addition to the above scheme, the way that XEmacs handles mapping the
initial frame should be changed.  However, this change perhaps should be
delayed to a later version of XEmacs because of the user visible changes
that it entails and the possible breakage in people's init files that
might occur. (For example, if the rest of the scheme is implemented in
21.2, then this part of the scheme might want to be delayed until
version 22.)  The basic idea is that the initial frame is not created
before the initialization file is run, but instead a banner frame is
created containing the XEmacs logo, a button that allows the user to
cancel the execution of the init file and an area where messages that
are output in the process of running this file are displayed.  This area
should contain a number of lines, which makes it better than the current
scheme where only the last message is visible.  After the init file is
done, the initial frame is mapped.  This way the init file can make face
changes and other such modifications that affect initial frame and then
have the initial frame correctly come up with these changes and not see
any frame dancing or other problems that exist currently.

   There should be a function that allows the initialization file to
explicitly create and map the first frame if it wants to.  There should
also be a pre-init property that controls whether the banner frame
appears (of course it defaults to true) a property controlling when the
initial frame is created (before or after the init file, defaulting to
after), and a property controlling whether the initial frame is mapped
(normally true, but will be false if the `-unmapped' command line
argument is given).

   If an error occurs in the init file, then the initial frame should
always be created and mapped at that time so that the error is displayed
and the debugger has a place to be invoked.


File: internals.info,  Node: Future Work -- Keyword Parameters,  Next: Future Work -- Property Interface Changes,  Prev: Future Work -- Better Initialization File Scheme,  Up: Future Work

44.6 Future Work - Keyword Parameters
=====================================

Author: Ben Wing (mailto:ben@xemacs.org)

   NOTE: These changes are partly motivated by the various
user-interface changes elsewhere in this document, and partly for Mule
support.  In general the various APIs in this document would benefit
greatly from built-in keywords.

   I would like to make keyword parameters an integral part of Elisp.
The idea here is that you use the `&amp;key' identifier in the
parameter list of a function and all of the following parameters
specified are keyword parameters.  This means that when these arguments
are specified in a function call, they are immediately preceded in the
argument list by a "keyword", which is a symbol beginning with the `:'
character.  This allows any argument to be specified independently of
any other argument with no need to place the arguments in any
particular order.  This is particularly useful for functions that take
many optional parameters; using keyword parameters makes the code much
cleaner and easier to understand.

   The `cl' package already provides keyword parameters of a sort, but
I would like to make this more integrated and useable in a standard
fashion.  The interface that I am proposing is essentially compatible
with the keyword interface in Common Lisp, but it may be a subset of the
Common Lisp functionality, especially in the first implementation.
There is one departure from the Common Lisp specification that I would
like to make in order to make it much easier to add keyword parameters
to existing functions with optional parameters, and in general, to make
optional and keyword parameters coexist more easily.  The Common Lisp
specification indicates that if a function has both optional and keyword
parameters, the optional parameters are always processed before the
keyword parameters.  This means, for example, that if a function has
three required parameters, two optional parameters, and some number of
keyword parameters following, and the program attempts to call this
function by passing in the three required arguments, and then some
keyword arguments, the first keyword specified and the argument
following it get assigned to the first and second optional parameters as
specified in the function definition.  This is certainly not what is
intended, and means that if a function defines both optional and keyword
parameters, any calls of this function must specify `nil' for all of
the optional arguments before using any keywords.  If the function
definition is later changed to add more optional parameters, all
existing calls to this function that use any keyword arguments will
break.  This problem goes away if we simply process keyword parameters
before the optional parameters.

   The primary changes needed to support the keyword syntax are:

  1.  The subr object type needs to be modified to contain additional
     slots for the number and names of any keyword parameters.

  2.  The implementation of the `funcall' function needs to be modified
     so that it knows how to process keyword parameters.  This is the
     only place that will require very much intricate coding, and much
     of the logic that would need to be added can be lifted directly
     from the `cl' code.

  3.  A new macro, similar to the `DEFUN' macro, and probably called
     `DEFUN_WITH_KEYWORDS', needs to be defined so that built-in Lisp
     primitives containing keywords can be created.  Now, the
     `DEFUN_WITH_KEYWORDS' macro should take an additional parameter
     which is a string, which consists of the part of the lambda list
     declaration for this primitive that begins with the `&amp;key'
     specifier.  This string is parsed in the `DEFSUBR' macro during
     XEmacs initialization, and is converted into the appropriate
     structure that needs to be stored into the subr object.  In
     addition, the MAX_ARGS parameter of the `DEFUN' macro needs to be
     incremented by the number of keyword parameters and these
     parameters are passed to the C function simply as extra parameters
     at the end.  The `DEFSUBR' macro can sort out the actual number of
     required, optional and keyword parameters that the function takes,
     once it has parsed the keyword parameter string.  (An alternative
     that might make the declaration of a primitive a little bit easier
     to understand would involve adding another parameter to the
     `DEFUN_WITH_KEYWORDS' macro that specifies the number of keyword
     parameters.  However, this would require some additional
     complexity in the preprocessor definition of the
     `DEFUN_WITH_KEYWORDS' macro, and probably isn't worth
     implementing).

  4.  The byte compiler would have to be modified slightly so that it
     knows about keyword parameters when it parses the parameter
     declaration of a function.  For example, so that it issues the
     correct warnings concerning calls to that function with incorrect
     arguments.

  5.  The `make-docfile' program would have to be modified so that it
     generates the correct parameter lists for primitives defined using
     the `DEFUN_WITH_KEYWORDS' macro.

  6.  Possibly other aspects of the help system that deal with function
     descriptions might have to be modified.

  7.  A helper function might need to be defined to make it easier for
     primitives that use both the `&amp;rest' and `&amp;key' specifiers
     to parse their argument lists.


Internal API for C primitives with keywords - necessary for many of the new Mule APIs being defined.
----------------------------------------------------------------------------------------------------

       DEFUN_WITH_KEYWORDS (Ffoo, "foo", 2, 5, 6, ALLOW_OTHER_KEYWORDS,
           (ichi, ARG_NIL), (ni, ARG_NIL), (san, ARG_UNBOUND), 0,
           (arg1, arg2, arg3, arg4, arg5)
           )
       {
         ...
       }

       -> C fun of 12 args:

       (arg1, ... arg5, ichi, ..., roku, other keywords)

       Circled in blue is actual example declaration

       DEFUN_WITH_KEYWORDS (Ffoo, "foo", 1,2,0 (bar, baz) <- arg list
       [ MIN ARGS, MAX ARGS, something that could be REST, SPECIFY_DEFAULT or
       REST_SPEC]

       [#KEYWORDS [ ALLOW_OTHER, SPECIFY_DEFAULT, ALLOW_OTHER_SPECIFY_DEFAULT
       6, ALLOW_OTHER_SPECIFY_DEFAULT,

       (ichi, 0) (ni, 0), (san, DEFAULT_UNBOUND), (shi, "t"), (go, "5"),
       (roku, "(current-buffer)")
       <- specifies arguments, default values (string to be read into Lisp
          data during init; then forms evalled at fn ref time.

       ,0 <- [INTERACTIVE SPEC] )

       LO = Lisp_Object

       -> LO Ffoo (LO bar, LO baz, LO ichi, LO ni, LO san, LO shi, LO go,
                   LO roku, int numkeywords, LO *other_keywords)

       #define DEFUN_WITH_KEYWORDS (fun, funstr, minargs, maxargs, argspec, \
                #args, num_keywords, keywordspec, keywords, intspec) \
       LO fun (DWK_ARGS (maxargs, args) \
               DWK_KEYWORDS (num_keywords, keywordspec, keywords))

       #define DWK_KEYWORDS (num_keywords, keywordspec, keywords) \
               DWK_KEYWORDS ## keywordspec (keywords)
               DWK_OTHER_KEYWORDS ## keywordspec)

       #define DWK_KEYWORDS_ALLOW_OTHER (x,y)
               DWK_KEYWORDS (x,y)

       #define DWK_KEYWORDS_ALLOW_OTHER_SPECIFICATIONS (x,y)
               DWK_KEYWORDS_SPECIFY_DEFAULT (x,y)

       #define DWK_KEYWORDS_SPECIFY_DEFAULT (numkey, key)
               ARGLIST_CAR ## numkey key

       #define ARGLT_GRZ (x,y) LO CAR x, LO CAR y


File: internals.info,  Node: Future Work -- Property Interface Changes,  Next: Future Work -- Toolbars,  Prev: Future Work -- Keyword Parameters,  Up: Future Work

44.7 Future Work - Property Interface Changes
=============================================

Author: Ben Wing (mailto:ben@xemacs.org)

   In my past work on XEmacs, I already expanded the standard property
functions of `get', `put', and `remprop' to work on objects other than
symbols and defined an additional function `object-plist' for this
interface.  I'd like to expand this interface further and advertise it
as the standard way to make property changes in objects, especially the
new objects that are going to be defined in order to support the added
user interface features of version 22.  My proposed changes are as
follows:

  1.  A new concept associated with each property called a "default
     value" is introduced.  (This concept already exists, but not in a
     well-defined way.) The default value is the value that the
     property assumes for certain value retrieval functions such as
     `get' when it is "unbound", which is to say that its value has not
     been explicitly specified. Note: the way to make a property
     unbound is to call `remprop'.  Note also that for some built-in
     properties, setting the property to its default value is
     equivalent to making it unbound.

  2.  The behavior of the `get' function is modified.  If the `get'
     function is called on a property that is unbound and the third,
     optional DEFAULT argument is `nil', then the default value of the
     property is returned.  If the DEFAULT argument is not `nil', then
     whatever was specified as the value of this argument is returned.
     For the most part, this is upwardly compatible with the existing
     definition of `get' because all user-defined properties have an
     initial default value of `nil'.  Code that calls the `get'
     function and specifies `nil' for the DEFAULT argument, and expects
     to get `nil' returned if the property is unbound, is almost
     certainly wrong anyway.

  3.  A new function, `get1' is defined.  This function does not take a
     default argument like the `get' function.  Instead, if the property
     is unbound, an error is signaled.  Note: `get' can be implemented
     in terms of `get1'.

  4.  New functions `property-default-value' and `property-bound-p' are
     defined with the obvious semantics.

  5.  An additional function `property-built-in-p' is defined which
     takes two arguments, the first one being a symbol naming an object
     type, and the second one specifying a property, and indicates
     whether the property name has a built-in meaning for objects of
     that type.

  6.  It is not necessary, or even desirable, for all object types to
     allow user-defined properties.  It is always possible to simulate
     user-defined properties for an object by using a weak hash table.
     Therefore, whether an object allows a user to define properties or
     not should depend on the meaning of the object.  If an object does
     not allow user-defined properties, the `put' function should
     signal an error, such as `undefined-property', when given any
     property other than those that are predefined.

  7.  A function called `user-defined-properties-allowed-p' should be
     defined with the obvious semantics.  (See the previous item.)

  8.  Three more functions should be defined, called
     `built-in-property-name-list', `property-name-list', and
     `user-defined-property-name-list'.


   Another idea:

     (define-property-method
       predicate object-type
       predicate cons :(KEYWORD)  (all lists beginning with KEYWORD)

       :put putfun
       :get
       :remprop
       :object-props
       :clear-properties
       :map-properties

       e.g. (define-property-method 'hash-table
              :put #'(lambda (obj key value) (puthash key obj value)))


File: internals.info,  Node: Future Work -- Toolbars,  Next: Future Work -- Menu API Changes,  Prev: Future Work -- Property Interface Changes,  Up: Future Work

44.8 Future Work - Toolbars
===========================

* Menu:

* Future Work -- Easier Toolbar Customization::
* Future Work -- Toolbar Interface Changes::


File: internals.info,  Node: Future Work -- Easier Toolbar Customization,  Next: Future Work -- Toolbar Interface Changes,  Prev: Future Work -- Toolbars,  Up: Future Work -- Toolbars

44.8.1 Future Work - Easier Toolbar Customization
-------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* One of XEmacs' greatest strengths is its ability to be
customized endlessly.  Unfortunately, it is often too difficult to
figure out how to do this.  There has been some recent work like the
Custom package, which helps in this regard, but I think there's a lot
more work that needs to be done.  Here are some ideas (which certainly
could use some more thought).

   Although there is currently an `edit-toolbar' package, it is not
well integrated with XEmacs, and in general it is much too hard to
customize the way toolbars look.  I would like to see an interface that
works a bit like the way things work under Windows, where you can
right-click on a toolbar to get a menu of options that allows you to
change aspects of the toolbar.  The general idea is that if you
right-click on an item itself, you can do things to that item, whereas
if you right-click on a blank part of a toolbar, you can change the
properties of the toolbar.  Some of the items on the right-click menu
for a particular toolbar button should be specified by the button
itself.  Others should be standard.  For example, there should be an
*Execute* item which simply does what would happen if you left-click on
a toolbar button.  There should probably be a *Delete* item to get rid
of the toolbar button and a *Properties* item, which brings up a
property sheet that allows you to do things like change the icon and
the command string that's associated with the toolbar button.

   The options to change the appearance of the toolbar itself should
probably appear both on the context menu for specific buttons, and on
the menu that appears when you click on a blank part of the toolbar.
That way, if there isn't a blank part of the toolbar, you can still
change the toolbar appearance.  As for what appears in these items, in
Outlook Express, for example, there are three different menu items, one
of which is called *Buttons*, which brings up, or pops up a window
which allows you to edit the toolbar, which for us could pop up a new
frame, which is running `edit-toolbar.el'.  The second item is called
*Align*, which contains a submenu that says *Top*, *Bottom*, *Left*,
and *Right*, which will be just like setting the default toolbar
position.  The third one says *Text Labels*, which would just let you
select whether there are captions or not.  I think all three of these
are useful and are easy to implement in XEmacs.  These things also need
to be integrated with custom so that a user can control whether these
options apply to all sessions, and in such a case can save the settings
out to an options file.  `edit-toolbar.el' in particular needs to
integrate with custom.  Currently it has some sort of hokey stuff of
its own, which it saves out to a `.toolbar' file.  Another useful
option to have, once we draw the captions dynamically rather than using
pre-generated ones, would be the ability to change the font size of the
captions.  I'm sure that Kyle, for one, would appreciate this.

   (This is incomplete.....)


File: internals.info,  Node: Future Work -- Toolbar Interface Changes,  Prev: Future Work -- Easier Toolbar Customization,  Up: Future Work -- Toolbars

44.8.2 Future Work - Toolbar Interface Changes
----------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   I propose changing the way that toolbars are specified to make them
more flexible.

  1.  A new format for the vector that specifies a toolbar item is
     allowed.  In this format, the first three items of the vector are
     required and are, respectively, a caption, a glyph list, and a
     callback.  The glyph list and callback arguments are the same as
     in the current toolbar item specification, and the caption is a
     string specifying the caption text placed below the toolbar glyph.
     The caption text is required so that toolbar items can be
     identified for the purpose of retrieving and changing their
     property values.  Putting the caption first also makes it easy to
     distinguish between the new and the old toolbar item vector
     formats.  In the old format, the first item, the glyph list, is
     either a list or a symbol.  In the new format, the first item is a
     string.  In the new format, following the three required items,
     are optional keyword items specified using keywords in the same
     format as the menu item vector format.  The keywords that should
     be predefined are: `:help-echo', `:context-menu',
     `:drop-handlers', and `:enabled-p'.  The `:enabled-p' and
     `:help-echo' keyword arguments are the same as the third and
     fourth items in the old toolbar item vector format.  The
     `:context-menu' keyword is a list in standard menu format that
     specifies additional items that will appear when the context menu
     for the toolbar item is popped up.  (Typically, this happens when
     the right mouse button is clicked on the toolbar item).  The
     `:drop-handlers' keyword is for use by the new drag-n-drop
     interface (see Drag-n-Drop Interface Changes (drag-n-drop.html) ),
     and is not normally specified or modified directly.

  2.  Conceivably, there could also be keywords that are associated
     with a toolbar itself, rather than with a particular toolbar item.
     These keyword properties would be specified using keywords and
     arguments that occur before any toolbar item vectors, similarly to
     how things are done in menu specifications.  Possible properties
     could include `:captioned-p' (whether the captions are visible
     under the toolbar), `:glyphs-visible-p' (whether the toolbar
     glyphs are visible), and `:context-menu' (additional items that
     will appear on the context menus for all toolbar items and
     additionally will appear on the context menu that is popped up
     when the right mouse button is clicked over a portion of the
     toolbar that does not have any toolbar buttons in it).  The
     current standard practice with regards to such properties seems to
     be to have separate specifiers, such as `left-toolbar-width',
     `right-toolbar-width', `left-toolbar-visible-p',
     `right-toolbar-visible-p', etc.  It could easily be argued that
     there should be no such toolbar specifiers and that all such
     properties should be part of the toolbar instantiator itself.  In
     this scheme, the only separate specifiers that would exist for
     individual properties would be default values.  There are a lot of
     reasons why an interface change like this makes sense.  For
     example, currently when VM sets its toolbar, it also sets the
     toolbar width and similar properties.  If you change which edge of
     the frame the VM toolbar occurs in, VM will also have to go and
     modify all of the position-specific toolbar specifiers for all of
     the other properties associated with a toolbar.  It doesn't really
     seem to make sense to me for the user to be specifying the width
     and visibility and such of specific toolbars that are attached to
     specific edges because the user should be free to move the
     toolbars around and expect that all of the toolbar properties
     automatically move with the toolbar. (It is also easy to imagine,
     for example, that a toolbar might not be attached to the edge of
     the frame at all, but might be floating somewhere on the user's
     screen).  With an interface where these properties are separate
     specifiers, this has to be done manually.  Currently, having the
     various toolbar properties be inside of toolbar instantiators
     makes them difficult to modify, but this will be different with
     the API that I propose below.

  3.  I propose an API for modifying toolbar and toolbar item
     properties, as well as making other changes to toolbar
     instantiators, such as inserting or deleting toolbar items.  This
     API is based around the concept of a path.  There are two kinds of
     paths here - "toolbar paths" and "toolbar item paths".  Each kind
     of path is an object (of type `toolbar-path' and
     `toolbar-item-path', respectively) whose properties specify the
     location in a toolbar instantiator where changes to the
     instantiator can be made.  A toolbar path, for example, would be
     created using the `make-toolbar-path' function, which takes a
     toolbar specifier (or optionally, a symbol, such as `left',
     `right', `default', or `nil', which refers to a particular
     toolbar), and optionally, parameters such as the locale and the
     tag set, which specify which actual instantiator inside of the
     toolbar specifier is to be modified.  A toolbar item path is
     created similarly using a function called
     `make-toolbar-item-path', which takes a toolbar specifier and a
     string naming the caption of the toolbar item to be modified, as
     well as, of course, optionally the locale and tag set parameters
     and such.

     The usefulness of these path objects is as arguments to functions
     that will use them as pointers to the place in a toolbar
     instantiator where the modification should be made.  Recall, for
     example, the generalized property interface described above.  If a
     function such as `get' or `put' is called on a toolbar path or
     toolbar item path, it will use the information contained in the
     path object to retrieve or modify a property located at the end of
     the path.  The toolbar path objects can also be passed to new
     functions that I propose defining, such as `add-toolbar-item',
     `delete-toolbar-item', and `find-toolbar-item'.  These functions
     should be parallel to the functions for inserting, deleting,
     finding, etc. items in a menu.  The toolbar item path objects can
     also be passed to the drop-handler functions defined in
     Drag-n-Drop Interface Changes (drag-n-drop.html) to retrieve or
     modify the drop handlers that are associated with a toolbar item.
     (The idea here is that you can drag an object and drop it onto a
     toolbar item, just as you could onto a buffer, an extent, a menu
     item, or any other graphical element).

  4.  We should at least think about allowing for separate default and
     buffer-local toolbars.  The user should either be able to position
     these toolbars one above the other, or side by side, occupying a
     single toolbar line.  In the latter case, the boundary between the
     toolbars should be draggable, and if a toolbar takes up more room
     than is allocated for it, there should be arrows that appear on
     one or both sides of the toolbar so that the items in the toolbar
     can be scrolled left or right.  (For that matter, this sort of
     interface should exist even when there is only one toolbar that is
     on a particular toolbar line, because the toolbar may very well
     have more items than can be displayed at once, and it's silly in
     such a case if it's impossible to access the items that are not
     currently visible).

  5.  The default context menu for toolbars (which should be specified
     using a specifier called `default-toolbar-context-menu' according
     to the rules defined above) should contain entries allowing the
     user to modify the appearance of a toolbar.  Entries would
     include, for example, whether the toolbar is captioned, whether
     the glyphs for the toolbar are visible (if the toolbar is
     captioned but its glyphs are not visible, the toolbar appears as
     nothing but text; you can set things up this way, for example, in
     Netscape), an option that brings up a package for editing the
     contents of a toolbar, an option to allow the caption face to be
     dchanged (perhaps thorough jan `edit-faces' or `custom'
     interface), etc.



File: internals.info,  Node: Future Work -- Menu API Changes,  Next: Future Work -- Removal of Misc-User Event Type,  Prev: Future Work -- Toolbars,  Up: Future Work

44.9 Future Work - Menu API Changes
===================================

Author: Ben Wing (mailto:ben@xemacs.org)

  1.  I propose making a specifier for the menubar associated with the
     frame.  The specifier should be called `default-menubar' and
     should replace the existing `current-menubar' variable.  This
     would increase the power of the menubar interface and bring it in
     line with the toolbar interface.  (In order to provide proper
     backward compatibility, we might have to complete the symbol value
     handler mechanism (symbol-value-handlers.html))

  2.  I propose an API for modifying menu instantiators similar to the
     API composed above for toolbar instantiators.  A new object called
     a "menu path" (of type `menu-path') can be created using the
     `make-menu-path' function, and specifies a location in a particular
     menu instantiator where changes can be made.  The first argument to
     `make-menu-path' specifies which menu to modify and can be a
     specifier, a value such as `nil' (which means to modify the default
     menubar associated with the selected frame), or perhaps some other
     kind of specification referring to some other menu, such as the
     context menus invoked by the right mouse button.  The second
     argument to `make-menu-path', also required, is a list of zero or
     more strings that specifies the particular menu or menu item in
     the instantiator that is being referred to.  The remaining
     arguments are optional and would be a locale, a tag set, etc.  The
     menu path object can be passed to `get', `put' or other standard
     property functions to access or modify particular properties of a
     menu or a menu item.  It can also be passed to expanded versions
     of the existing functions such as `find-menu-item',
     `delete-menu-item', `add-menu-button', etc.  (It is really a shame
     that `add-menu-item' is an obsolete function because it is a much
     better name than `add-menu-button').  Finally, the menu path
     object can be passed to the drop-handler functions described in
     Drag-n-Drop Interface Changes (drag-n-drop.html) to access or
     modify the drop handlers that are associated with a particular
     menu item.

  3.  New keyword properties should be added to the menu item vector.
     These include `:help-echo', `:context-menu' and `:drop-handlers',
     with similar semantics to the corresponding keywords for toolbar
     items.  (It may seem a bit strange at first to have a context menu
     associated with a particular menu item, but it is a user interface
     concept that exists both in Open Look and in Windows, and really
     makes a lot of sense if you give it a bit of thought).  These
     properties may not actually be implemented at first, but at least
     the keywords for them should be defined.



File: internals.info,  Node: Future Work -- Removal of Misc-User Event Type,  Next: Future Work -- Mouse Pointer,  Prev: Future Work -- Menu API Changes,  Up: Future Work

44.10 Future Work - Removal of Misc-User Event Type
===================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* This page describes why the misc-user event type should
be split up into a number of different event types, and how to do this.

   The misc-user event should not exist as a single event type.  It
should be split up into a number of different event types: one for
scrollbar events, one for menu events, and one or two for drag-n-drop
events.  Possibly there will be other event types created in the
future.  The reason for this is that the misc-user event was a bad
design choice when I made it, and it has only gotten worse with
Oliver's attempts to add features to it to make it be used for
drag-n-drop.  I know that there was originally a separate drag-n-drop
event type, and it was folded into the misc-user event type on my
recommendation, but I have now realized the error of my ways.  I had
originally created a single event type in an attempt to prevent some
Lisp programs from breaking because they might have a case statement
over various event types, and would not be able to handle new event
types appearing.  I think now that these programs simply need to be
written in a way to handle new event types appearing.  It's not very
hard to do this.  You just use predicates instead of doing a case
statement over the event type.  If we preserve the existing predicate
called `misc-user-event-p', and just make sure that it evaluates to
true when given any user event type other than the standard simple
ones, then most existing code will not break either when we split the
event types up like this, or if we add any new event types in the
future.

   More specifically, the only clean way to design the misc-user event
type would be to add a sub-type field to it, and then have the nature
of all the other fields in the event type be dependent on this
sub-type.  But then in essence, we'd just be reimplementing the whole
event-type scheme inside of misc-user events, which would be rather
pointless.


File: internals.info,  Node: Future Work -- Mouse Pointer,  Next: Future Work -- Version Number and Development Tree Organization,  Prev: Future Work -- Removal of Misc-User Event Type,  Up: Future Work

44.11 Future Work - Mouse Pointer
=================================

* Menu:

* Future Work -- Abstracted Mouse Pointer Interface::
* Future Work -- Busy Pointer::


File: internals.info,  Node: Future Work -- Abstracted Mouse Pointer Interface,  Next: Future Work -- Busy Pointer,  Prev: Future Work -- Mouse Pointer,  Up: Future Work -- Mouse Pointer

44.11.1 Future Work - Abstracted Mouse Pointer Interface
--------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* We need to create a new image format that allows
standard pointer shapes to be specified in a way that works on all
Windows systems.  I suggest that this be called `pointer', which has
one tag associated with it, named `:data', and whose value is a string.
The possible strings that can be specified here are predefined by
XEmacs, and are guaranteed to work across all Windows systems.  This
means that we may need to provide our own definition for pointer shapes
that are not standard on some systems.  In particular, there are a lot
more standard pointer shapes under X than under Windows, and most of
these pointer shapes are fairly useful.  There are also a few pointer
shapes (I think the hand, for example) on Windows, but not on X.
Converting the X pointer shapes to Windows should be easy because the
definitions of the pointer shapes are simply XBM files, which we can
read under Windows.  Going the other way might be a little bit more
difficult, but it should still not be that hard.

   While we're at it, we should change the image format currently called
`cursor-font' to `x-cursor-font', because it only works under X
Windows.  We also need to change the format called `resource' to be
`mswindows-resource'.  At least in the case of `cursor-font', the old
value should be maintained for compatibility as an obsolete alias.  The
`resource' format was added so recently that it's possible that we can
just change it.


File: internals.info,  Node: Future Work -- Busy Pointer,  Prev: Future Work -- Abstracted Mouse Pointer Interface,  Up: Future Work -- Mouse Pointer

44.11.2 Future Work - Busy Pointer
----------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Automatically make the mouse pointer switch to a busy shape (watch
signal) when XEmacs has been "busy" for more than, e.g. 2 seconds.
Define the "busy time" as the time since the last time that XEmacs was
ready to receive input from the user.  An implementation might be:

  1. Set up an asynchronous timeout, to signal after the busy time;
     these are triggered through a call to QUIT so they will be
     triggered even when the code is busy doing something.

  2. We already have an "emacs_is_blocking" flag when we are waiting for
     input.  In the same place, when we are about to block and wait for
     input (regardless of whether input is already present), maybe call
     a hook, which in this case would remove the timer and put back the
     normal mouse shape.  Then when we exit the blocking stage (we got
     some input), call another hook, which in this case will start the
     timer.  Note that we don't want these "blocking" hooks to be
     triggered just because of an accept-process-output or some similar
     thing that retrieves events, only to put them back onto a queue
     for later processing.  Maybe we want some sort of flag that's
     bound by those routines saying that we aren't really waiting for
     input.  Making that flag Lisp-accessible allows it to be set by
     similar sorts of Lisp routines (if there are any?) that loop
     retrieving events but defer them, or only drain the queue, or
     whatnot.  #### Think about whether it would make some sense to try
     and be more clever in our determinations of what counts as "real
     waiting for user input", e.g.  whether the event gets dispatched
     (unfortunately this occurs way too late, we want to know to remove
     the busy cursor *before* getting an event), maybe whether there
     are any events waiting to be processed or we'll truly block, etc.
     (e.g. one possibility if there is input on the queue already when
     we "block" for input, don't remove the busy- wait pointer, but
     trigger the removal of it when we dispatch a user event).


File: internals.info,  Node: Future Work -- Version Number and Development Tree Organization,  Next: Future Work -- Improvements to the `xemacs.org' Website,  Prev: Future Work -- Mouse Pointer,  Up: Future Work

44.12 Future Work - Version Number and Development Tree Organization
====================================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* The purpose of this proposal is to present a coherent
plan for how development branches in XEmacs are managed.  This will
cover such issues as stable versus experimental branches, creating new
branches, synchronizing patches between branches, and how version
numbers are assigned to branches.

   A development branch is defined to be a linear series of releases of
the XEmacs code base, each of which is derived from the previous one.
When the XEmacs development tree is forked and two branches are created
where there used to be one, the branch that is intended to be more
stable and have fewer changes made to it is considered the one that
inherits the parent branch, and the other branch is considered to have
begun at the branching point.  The less stable of the two branches will
eventually be forked again, while this will not happen usually to the
more stable of the two branches, and its development will eventually
come to an end.  This means that every branch has a definite ending
point.  For example, the 20.x branch began at the point when the
released 19.13 code tree was split into a 19.x and a 20.x branch, and a
20.x branch will end when the last 20.x release (probably numbered 20.5
or 20.6) is released.

   I think that there should always be three active development
branches at any time.  These branches can be designated the stable, the
semi-stable, and the experimental branches.  This situation has existed
in the current code tree as soon as the 21.0 development branch was
split.  In this situation, the stable branch is the 20.x series.  The
semi-stable branch is the 21.0 release and the stability releases that
follow.  The experimental branch is the branch that was created as the
result of the 21.0 development branch split.  Typically, the stable
branch has been released for a long period of time.  The semi-stable
branch has been released for a short period of time, or is about to be
released, and the experimental branch has not yet been released, and
will probably not be released for awhile.  The conditions that should
hold in all circumstances are:

  1.  There should be three active branches.

  2.  The experimental branch should never be in feature freeze.


   The reason for the second condition is to ensure that active
development can always proceed and is never throttled, as is happening
currently at the end of the 21.0 release cycle.  What this means is
that as soon as the experimental branch is deemed to be stable enough
to go into feature freeze:

  1.  The current stable branch is made inactive and all further
     development on it ceases.

  2.  The semi-stable branch, which by now should have been released
     for a fair amount of time, and should be fairly stable, gets
     renamed to the stable branch.

  3.  The experimental branch is forked into two branches, one of which
     becomes the semi-stable branch, and the other, the experimental
     branch.


   The stable branch is always in high resistance, which is to say that
the only changes that can be made to the code are important bug fixes
involving a small amount of code where it should be clear just by
reading the code that no destabilizing code has been introduced.  The
semi-stable branch is in low resistance, which means that no major
features can be added, but except right before a release fairly major
code changes are allowed.  Features can be added if they are
sufficiently small, if they are deemed sufficiently critical due to
severe problems that would exist if the features were not added (for
example, replacement of the unexec mechanism with a portable solution
would be a feature that could be added to the semi-stable branch
provided that it did not involve an overly radical code re-architecture,
because otherwise it might be impossible to build XEmacs on some
architectures or with some compilers), or if the primary purpose of the
new feature is to remedy an incompleteness in a recent architectural
change that was not finished in a prior release due to lack of time (for
example, abstracting the mouse pointer and list-of-colors interfaces,
which were left out of 21.0).  There is no feature resistance in place
in the experimental branch, which allows full development to proceed at
all times.

   In general, both the stable and semi-stable branches will contain
previous net releases.  In addition, there will be beta releases in all
three branches, and possibly development snapshots between the beta
releases.  It's obviously necessary to have a good version numbering
scheme in order to keep everything straight.

   First of all, it needs to be immediately clear from the version
number whether the release is a beta release or a net release.  Steve
has proposed getting rid of the beta version numbering system, which I
think would be a big mistake.  Furthermore, the net release version
number and beta release version number should be kept separate, just as
they are now, to make it completely clear where any particular release
stands.  There may be alternate ways of phrasing a beta release other
than something like 21.0 beta 34, but in all such systems, the beta
number needs to be zero for any release version.  Three possible
alternative systems, none of which I like very much, are:

  1.  The beta number is simply an extra number in the regular version
     number.  Then, for example, 21.0 beta 34 becomes 21.0.34.  The
     problem is that the release version, which would simply be called
     21.0, appears to be earlier than 21.0 beta 34.

  2.  The beta releases appear as later revisions of earlier releases.
     Then, for example, 21.1 beta 34 becomes 21.0.34, and 21.0 beta 34
     would have to become 21.-1.34.  This has both the obvious ugliness
     of negative version numbers and the problem that it makes beta
     releases appear to be associated with their previous releases,
     when in fact they are more closely associated with the following
     release.

  3.  Simply make the beta version number be negative.  In this scheme,
     you'd start with something like -1000 as the first beta, and then
     21.0 beta 34 would get renumbered to 21.0.-968.  Obviously, this
     is a crazy and convoluted scheme as well, and we would be best to
     avoid it.


   Currently, the between-beta snapshots are not numbered, but I think
that they probably should be.  If appropriate scripts are handled to
automate beta release, it should be very easy to have a version number
automatically updated whenever a snapshot is made.  The number could be
added either as a separate snapshot number, and you'd have 21.0 beta 34
pre 1, which becomes before 21.0 beta 34; or we could make the beta
number be floating point, and then the same snapshot would have to be
called 21.0 beta 33.1.  The latter solution seems quite kludgey to me.

   There also needs to be a clear way to distinguish, when a net
release is made, which branch the release is a part of.  Again, three
solutions come to mind:

  1.  The major version number reflects which development branch the
     release is in and the minor version number indicates how many
     releases have been made along this branch.  In this scheme, 21.0
     is always the first release of the 21 series development branch,
     and when this branch is split, the child branch that becomes the
     experimental branch gets version numbers starting with 22.  This
     scheme is the simplest, and it's the one I like best.

  2.  We move to a three-part version number.  In this scheme, the
     first two numbers indicate the branch, and the third number
     indicates the release along the branch.  In this scheme, we have
     numbers like 21.0.1, which would be the second release in the 21.0
     series branch, and 21.1.2, which would be the third release in the
     21.1 series branch.  The major version number then gets increased
     only very occasionally, and only when a sufficiently major
     architectural change has been made, particularly one that causes
     compatibility problems with code written for previous branches.  I
     think schemes like this are unnecessary in most circumstances,
     because usually either the major version number ends up changing
     so often that the second number is always either zero or one, or
     the major version number never changes, and as such becomes
     useless.  By the time the major version number would change, the
     product itself has changed so much that it often gets renamed.
     Furthermore, it is clear that the two version number scheme has
     been used throughout most of the history of Emacs, and recently we
     have been following the two number scheme also.  If we introduced a
     third revision number, at this point it would both confuse
     existing code that assumed there were two numbers, and would look
     rather silly given that the major version number is so high and
     would probably remain at the same place for quite a long time.

  3.  A third scheme that would attempt to cross the two schemes would
     keep the same concept of major version number as for the three
     number scheme, and would compress the second and third numbers of
     the three number scheme into one number by using increments of
     ten.  For example, the current 21.x branch would have releases No.
     21.0, 21.1, etc.  The next branch would be No. 21.10, 21.11, etc.
     I don't like this scheme very much because it seems rather
     kludgey, and also because it is not used in any other product as
     far as I know.

  4.  Another scheme that would combine the second and third numbers in
     the three number scheme would be to have the releases in the
     current 21.x series be numbered 21.0, then 21.01, then 22.02, etc.
     The next series is 21.1, then 21.11, then 21.12, etc.  This is
     similar to the way that version numbers are done for DOS in
     Windows.  I also think that this scheme is fairly silly because,
     like the previous scheme, its only purpose is to avoid increasing
     the major version number very much.  But given that we have
     already have a fairly large major version number, there doesn't
     seem to be any particular problem with increasing this number by
     one every year or two.  Some people will object that by doing
     this, it becomes impossible to tell when a change is so major that
     it causes a lot of code breakage, but past releases have not been
     accurate indicators of this.  For example, 19.12 caused a lot of
     code breakage, but 20.0 caused less, and 21.0 caused less still.
     In the GNU Emacs world, there were byte code changes made between
     19.28 and 19.29, but as far as I know, not between 19.29 and 20.0.


   With three active development branches, synchronizing code changes
between the branches is obviously somewhat of a problem.  To make things
easier, I propose a few general guidelines:

  1.  Merging between different branches need not happen that often.  It
     should not happen more often than necessary to avoid undue burden
     on the maintainer, but needs to be done at all defined
     checkpoints.  These checkpoints need to be noted in all of the
     places that track changes along the branch, for example, in all of
     the change logs and in all of the CVS tags.

  2.  Every code change that can be considered a self-contained unit, no
     matter how large or small, needs to have a change log entry,
     preferably a single change log entry associated with it.  This is
     an absolute requirement.  There should be no code changes without
     an associated change log entry.  Otherwise, it is highly likely
     that patches will not be correctly synchronized across all
     versions, and will get lost.  There is no need for change log
     entries to contain unnecessary detail though, and it is important
     that there be no more change log entries than necessary, which
     means that two or more change log entries associated with a single
     patch need to be grouped together if possible.  This might imply
     that there should be one global change log instead of change logs
     in each directory, or at the very least, the number of separate
     change logs should be kept to a minimum.

  3.  The patch that is associated with each change log entry needs to
     be kept around somewhere.  The reason for this is that when
     synchronizing code from some branch to some earlier branch, it is
     necessary to go through each change log entry and decide whether a
     change is worthy to make it into a more stable branch.  If so, the
     patch associated with this change needs to be individually applied
     to the earlier branch.

  4.  All changes made in more stable branches get merged into less
     stable branches unless the change really is completely unnecessary
     in the less stable branch because it is superseded by some other
     change.  This will probably mean more developers making changes to
     the semi-stable branch than to the experimental branch.  This
     means that developers should strive to do their development in the
     most stable branch that they expect their code to go into.  An
     alternative to this which is perhaps more workable is simply to
     insist that all developers make all patches based off of the
     experimental branch, and then later merge these patches down to
     the more stable branches as necessary.  This means, however, that
     submitted patches should never be combinations of two or more
     unrelated changes.  Whenever such patches are submitted, they
     should either be rejected (which should apply to anybody who
     should know better, which probably means everybody on the beta
     list and anybody else who is a regular contributor), or the
     maintainer or some other designated party needs to filter the
     combined patch into separate patches, one per logical change.

  5.  The maintainer should keep all the patches around in some data
     base, and the patches should be given an identifier consisting of
     the author of the patch, the date the patch was submitted, and
     some other identifying characteristic, such as a number, in case
     there is more than one patch on the same date by the same author.
     The database should hopefully be correctly marked at all times
     with something indicating which branches the patch has been
     applied to, and this database should hopefully be publicly visible
     so that patch authors can determine whether their patches have
     been applied, and whether their patches have been received, so
     that patches do not get needlessly resubmitted.

  6.  Global automatable changes such as textual renaming, reordering,
     and additions or deletions of parameters in function calls should
     still be allowed, even with multiple development branches.
     (Sometimes these are necessary for code cleanliness, and in the
     long run, they save a lot of time, even through they may cause
     some headaches in the short-term.)  In general, when such changes
     are made, they should occur in a separate beta version that
     contains only such changes and no other patches, and the changes
     should be made in both the semi-stable and experimental branches
     at the same time.  The description of the beta version should make
     it very clear that the beta is comprised of such changes.  The
     reason for doing these things is to make it easier for people to
     diff between beta versions in order to figure out the changes that
     were made without the diff getting cluttered up by these code
     cleanliness changes that don't change any actual behavior.



File: internals.info,  Node: Future Work -- Improvements to the `xemacs.org' Website,  Next: Future Work -- Keybindings,  Prev: Future Work -- Version Number and Development Tree Organization,  Up: Future Work

44.13 Future Work - Improvements to the `xemacs.org' Website
============================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   The `xemacs.org' web site is the face that XEmacs presents to the
outside world.  In my opinion, its most important function is to present
information about XEmacs in such a way that solicits new XEmacs users
and co-contributors.  Existing members of the XEmacs community can
probably find out most of the information they want to know about XEmacs
regardless of what shape the web site is in, or for that matter, perhaps
even if the web site doesn't exist at all.  However, potential new users
and co-contributors who go to the XEmacs web site and find it out of
date and/or lacking the information that they need are likely to be
turned away and may never return.  For this reason, I think it's
extremely important that the web site be up-to-date, well-organized, and
full of information that an inquisitive visitor is likely to want to
know.

   The current XEmacs web site needs a lot of work if it is to meet
these standards.  I don't think it's reasonable to expect one person to
do all of this work and make continual updates as needed, especially
given the dismal record that the XEmacs web site has had.  The proper
thing to do is to place the web site itself under CVS and allow many of
the core members to remotely check files in and out.  This way, for
example, Steve could update the part of the site that contains the
current release status of XEmacs. (Much of this could be done by a
script that Steve executes when he sends out a beta release
announcement which automatically HTML-izes the mail message and puts it
in the appropriate place on the web site.  There are programs that are
specifically designed to convert email messages into HTML, for example
`mhonarc'.)  Meanwhile, the `xemacs.org' mailing list administrator
(currently Jason Mastaler, I think) could maintain the part of the site
that describes the various mailing lists and other addresses at
`xemacs.org'.  Someone like me (perhaps through a proxy typist) could
maintain the part of the site that specifies the future directions that
XEmacs is going in, etc., etc.

   Here are some things that I think it's very important to add to the
web site.

  1.  A page describing in detail how to get involved in the XEmacs
     development process, how to submit and where to submit various
     patches to the XEmacs core or associated packages, how to contact
     the maintainers and core developers of XEmacs and the maintainers
     of various packages, etc.

  2.  A page describing exactly how to download, compile, and install
     XEmacs, and how to download and install the various binary
     distributions.  This page should particularly cover in detail how
     exactly the package system works from an installation standpoint
     and how to correctly compile and install under Microsoft Windows
     and Cygwin.  This latter section should cover what compilers are
     needed under Microsoft Windows and Cygwin, and how to get and
     install the Cygwin components that are needed.

  3.  A page describing where to get the various ancillary libraries
     that can be linked with XEmacs, such as the JPEG, TIFF, PNG,
     X-Face, DBM, and other libraries.  This page should also cover how
     to correctly compile it and install these libraries, including
     under Microsoft Windows (or at least it should contain pointers to
     where this information can be found).  Also, it should describe
     anything that needs to be specified as an option to `configure' in
     order for XEmacs to link with and make use of these libraries or
     of Motif or CDE.  Finally, this page should list which versions of
     the various libraries are required for use with the various
     different beta versions of XEmacs.  (Remember, this can change
     from beta to beta, and someone needs to keep a watchful eye on
     this).

  4.  Pointers to any other sites containing information on XEmacs.
     This would include, for example, Hrvoje's XEmacs on Windows FAQ
     and my Architecting XEmacs web site.  (Presumably, most of the
     information in this section will be temporary.  Eventually, these
     pages should be integrated into the main XEmacs web site).

  5.  A page listing the various sub-projects in the XEmacs development
     process and who is responsible for each of these sub-projects, for
     example development of the package system, administration of the
     mailing lists, maintenance of stable XEmacs versions, maintenance
     of the CVS web interface, etc.  This page should also list all of
     the packages that are archived at `xemacs.org' and who is the
     maintainer or maintainers for each of these packages.


Other Places with an XEmacs Presence
------------------------------------

We should try to keep an XEmacs presence in all of the major places on
the web that are devoted to free software or to the "open source"
community.  This includes, for example, the open source web site at
http://opensource.oreilly.com (../../opensource.oreilly.com/default.htm)
(I'm already in the process of contacting this site), the Freshmeat site
at http://www.freshmeat.net (../../www.freshmeat.net/default.htm), the
various announcement news groups (for example, comp.os.linux.announce
(news:comp.os.linux.announce), and the Windows announcement news group)
etc.


File: internals.info,  Node: Future Work -- Keybindings,  Next: Future Work -- Byte Code Snippets,  Prev: Future Work -- Improvements to the `xemacs.org' Website,  Up: Future Work

44.14 Future Work - Keybindings
===============================

* Menu:

* Future Work -- Keybinding Schemes::
* Future Work -- Better Support for Windows Style Key Bindings::
* Future Work -- Misc Key Binding Ideas::


File: internals.info,  Node: Future Work -- Keybinding Schemes,  Next: Future Work -- Better Support for Windows Style Key Bindings,  Prev: Future Work -- Keybindings,  Up: Future Work -- Keybindings

44.14.1 Future Work - Keybinding Schemes
----------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* We need a standard mechanism that allows a different
global key binding schemes to be defined.  Ideally, this would be the
keyboard action interface (keyboard-actions.html) that I have proposed,
however this would require a lot of work on the part of mode
maintainers and other external Elisp packages and will not be rady in
the short term.  So I propose a very kludgy interface, along the lines
of what is done in Viper currently.  Perhaps we can rip that key munging
code out of Viper and make a separate extension that implements a global
key binding scheme munging feature.  This way a key binding scheme could
rearrange all the default keys and have all sorts of other code, which
depends on the standard keys being in their default location, still
work.


File: internals.info,  Node: Future Work -- Better Support for Windows Style Key Bindings,  Next: Future Work -- Misc Key Binding Ideas,  Prev: Future Work -- Keybinding Schemes,  Up: Future Work -- Keybindings

44.14.2 Future Work - Better Support for Windows Style Key Bindings
-------------------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* This page describes how we could create an XEmacs
extension that modifies the global key bindings so that a Windows user
would feel at home when using the keyboard in XEmacs.  Some of these
bindings don't conflict with standard XEmacs keybindings and should be
added by default, or at the very least under Windows, and probably under
X Windows as well. Other key bindings would need to be implemented in a
Windows compatibility extension which can be enabled and disabled on the
fly, following the conventions outlined in Standard interface for
enabling extensions (enabling-extensions.html) Ideally, this should be
implemented using the keyboard action interface (keyboard-actions.html)
but these wil not be available in the short term, so we will have to
resort to some awful kludges, following the model of Michael Kifer's
Viper mode.

   We really need to make XEmacs provide standard Windows key bindings
as much as possible.  Currently, for example, there are at least two
packages that allow the user to make a selection using the shifted arrow
keys, and neither package works all that well, or is maintained.  There
should be one well-written piece of code that does this, and it should
be a standard part of XEmacs.  In fact, it should be turned on by
default under Windows, and probably under X as well. (As an aside here,
one point of contention in how to implement this involves what happens
if you select a region using the shifted arrow keys and then hit the
regular arrow keys.  Does the region remain selected or not?  I think
there should be a variable that controls which of these two behaviors
you want.  We can argue over what the default value of this variable
should be.  The standard Windows behavior here is to keep the region
selected, but move the insertion point elsewhere, which is unfortunately
impossible to implement in XEmacs.)

   Some thought should be given to what to do about the standard Windows
control and alt key bindings.  Under NTEmacs, there is a variable that
controls whether the alt key behaves like the Emacs meta key, or whether
it is passed on to the menu as in standard Windows programs.  We should
surely implement this and put this option on the *Options* menu.
Making `Alt-f' for example, invoke the *File* menu, is not all that
disruptive in XEmacs, because the user can always type `ESC f' to get
the meta key functionality.  Making `Control-x', for example, do *Cut*,
is much, much more problematic, of course, but we should consider how
to implement this anyway.  One possibility would be to move all of the
current Emacs control key bindings onto control-shift plus a key, and
to make the simple control keys follow the Windows standard as much as
possible.  This would mean, for example, that we would have the
following keybindings:
`Control-x' ==> *Cut*
`Control-c' ==> *Copy*
`Control-v' ==> *Paste*
`Control-z' ==> *Undo*
`Control-f' ==> *Find*
`Control-a' ==> *Select All*
`Control-s' ==> *Save*
`Control-p' ==> *Print*
`Control-y' ==> *Redo*
(this functionality _is_ available in XEmacs with Kyle Jones' `redo.el'
package, but it should be better integrated)
`Control-n' ==> *New*
`Control-o' ==> *Open*
`Control-w' ==> *Close Window*
The changes described in the previous paragraph should be put into an
extension named `windows-keys.el' (see Standard interface for enabling
extensions (enabling-extensions.html)) so that it can be enabled and
disabled on the fly using a menu item and can be selected as the
default for a particular user in their custom options file. Once this
is implemented, the Windows installer should also be modified so that
it brings up a dialog box that allows the user to make a selection of
which key binding scheme they would prefer as the default, either the
XEmacs standard bindings, Vi bindings (which would be Viper mode),
Windows-style bindings, Brief, CodeWright, Visual C++, or whatever we
manage to implement.


File: internals.info,  Node: Future Work -- Misc Key Binding Ideas,  Prev: Future Work -- Better Support for Windows Style Key Bindings,  Up: Future Work -- Keybindings

44.14.3 Future Work - Misc Key Binding Ideas
--------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   * M-123 ... do digit arg

   * However, M-( group commands together until M-)

   * Nested M-() are allowed.

   * Number repeating plus () repeats N times each group of commands as
     a unit.

   * M-() by itself forms an anonymous macro, and there should be a
     command to repeat, like VI (execute macro), but when no () before,
     it repeats the last command of same amount of complication - or
     more like, somewhere there is a repeats all command back to make
     to act that stopping like VI's dot command.

   * C-numbers switches to a particular window.  maybe 1-3 or 1-4 does
     this.

   * C-4 or 5 to 9 (or ()? maybe reserved) switches to a particular
     frame.

   * Possibly C-Sh-numbers select more windows or frames.

   * M-C-1 M-C-2 M-C-3 M-C-4 M-C-5 M-C-6 M-C-7 M-C-8 M-C-9 M-C-0

     maybe should be execute anonymous macros (other possibility is
     insert register but you can easily simulate with a keyboard macro)

   * What about C-S M-C-S M-S??

   * I think there should be default fun key binding for *ILLEGIBLE*
     similar to what I have - load, save, cut, copy, paste, kill line,
     start/end macro, do macro


File: internals.info,  Node: Future Work -- Byte Code Snippets,  Next: Future Work -- Lisp Stream API,  Prev: Future Work -- Keybindings,  Up: Future Work

44.15 Future Work - Byte Code Snippets
======================================

Author: Ben Wing (mailto:ben@xemacs.org)

   * For use in time critical (e.g. redisplay) places such as display
     tables - a simple piece of code is evalled, e.g.
          (int-to-char (1+ c))
     where c is the arg, specbound.

   * can be compiled like
          (byte-compile-snippet (int-to-char (1+ c)) (c))
                                                     ^^^
                                          environment of local vars

   * need eval with bindings (not hard to implement) (extendable when
     lexical scoping present)

   * What's the return value of byte-compile-snippet?  (Look to see how
     this might be implemented)

* Menu:

* Future Work -- Autodetection::
* Future Work -- Conversion Error Detection::
* Future Work -- Unicode::
* Future Work -- BIDI Support::
* Future Work -- Localized Text/Messages::


File: internals.info,  Node: Future Work -- Autodetection,  Next: Future Work -- Conversion Error Detection,  Prev: Future Work -- Byte Code Snippets,  Up: Future Work -- Byte Code Snippets

44.15.1 Future Work - Autodetection
-----------------------------------

There are various proposals contained here.

New Implementation of Autodetection Mechanism
---------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   The current auto detection mechanism in XEmacs Mule has many
problems. For one thing, it is wrong too much of the time. Another
problem, although easily fixed, is that priority lists are fixed rather
than varying, depending on the particular locale; and finally, it
doesn't warn the user when it's not sure of the encoding or when there's
a mistake made during decoding. In both of these situations the user
should be presented with a list of likely encodings and given the
choice, rather than simply proceeding anyway and giving a result that is
likely to be wrong and may result in data corruption when the file is
saved out again.

   All coding systems are categorized according to their type. Currently
this includes ISO2022, Big 5, Shift-JIS, UTF8 and a few others. In the
future there will be many more types defined and this mechanism will be
generalized so that it is easily extendable by the Lisp programmer.

   In general, each coding system type defines a series of subtypes
which are handled differently for the purpose of detection. For
example, ISO 2022 defines many different subtypes such as 7 bit, 8 bit,
locking shift, designating and so on. UCS2 may define subtypes such as
normal and byte reversed.

   The detection engine works conceptually by calling the detection
methods of all of the defined coding system types in parallel on
successive chunks of data (which may, for example, be 4K in size, but
where the size makes no difference except for optimization purposes)
and watching the results until either a definite answer is determined
or the end of data is reached. The way the definite answer is
determined will be defined below. The detection method of the coding
system type is passed some data and a chunk of memory, which the method
uses to store its current state (and which is maintained separately for
each coding system type by the detection engine between successive
calls to the coding system type's detection method). Its return value
should be an alist consisting of a list of all of the defined subtypes
for that coding system type along with a level of likelihood and a list
of additional properties indicating certain features detected in the
data. The extra properties returned are defined entirely by the
particular coding system type and are used only in the algorithm
described below under "user control." However, the levels of likelihood
have a standard meaning as follows:

   Level 4 means "near certainty" and typically indicates that a
signature has been detected, usually at the beginning of the data,
indicating that the data is encoded in this particular coding system
type. An example of this would be the byte order mark at the beginning
of UCS2 encoded data or the GZIP mark at the beginning of GZIP data.

   Level 3 means "highly likely" and indicates that tell-tale signs have
been discovered in the data that are characteristic of this particular
coding system type. Examples of this might be ISO 2022 escape sequences
or the current Unicode end of line markers at regular intervals.

   Level 2 means "strongly statistically likely" indicating that
statistical analysis concludes that there's a high chance that this
data is encoded according to this particular type. For example, this
might mean that for UCS2 data, there is a high proportion of null bytes
or other repeated bytes in the odd-numbered bytes of the data and a
high variance in the even-numbered bytes of the data. For Shift-JIS,
this might indicate that there were no illegal Shift-JIS sequences and
a fairly high occurrence of common Shift-JIS characters.

   Level 1 means "weak statistical likelihood" meaning that there is
some indication that the data is encoded in this coding system type. In
fact, there is a reasonable chance that it may be some other type as
well. This means, for example, that no illegal sequences were
encountered and at least some data was encountered that is purposely
not in other coding system types. For Shift-JIS data, this might mean
that some bytes in the range 128 to 159 were encountered in the data.

   Level 0 means "neutral" which is to say that there's either not
enough data to make any decision or that the data could well be
interpreted as this type (meaning no illegal sequences), but there is
little or no indication of anything particular to this particular type.

   Level -1 means "weakly unlikely" meaning that some data was
encountered that could conceivably be part of the coding system type
but is probably not. For example, successively long line-lengths or
very rarely-encountered sequences.

   Level -2 means "strongly unlikely" meaning that typically a number
of illegal sequences were encountered.

   The algorithm to determine when to stop and indicate that the data
has been detected as a particular coding system uses a priority list,
which is typically specified as part of the language environment
determined from the current locale or the user's choice. This priority
list consists of a list of coding system subtypes, along with a minimum
level required for positive detection and optionally additional
properties that need to be present. Using the return values from all of
the detection methods called, the detection engine looks through this
priority list until it finds a positive match. In this priority list,
along with each subtype is a particular coding system to return when
the subtype is encountered. (For example, in a Japanese-language
environment particular subtypes of ISO 2022 will be associated with the
Japanese coding system version of those subtypes). It is perfectly
legal and quite common in fact, to list the same subtype more than once
in the priority list with successively lower requirements. Other facts
that can be listed in the priority list for a subtype are "reject",
meaning that the data should never be detected as this subtype, or
"ask", meaning that if the data is detected to be this subtype, the
user will be asked whether they actually mean this. This latter
property could be used, for example, towards the bottom of the priority
list.

   In addition there is a global variable which specifies the minimum
number of characters required before any positive match is reported.
There may actually be more than one such variable for different sources
of data, for example, detection of files versus detection of subprocess
data.

   Whenever a file is opened and detected to be a particular coding
system, the subtype, the coding system and the associated level of
likelihood will be prominently displayed either in the echo area or in
a status box somewhere.

   If no positive match is found according to the priority list, or if
the matches that are found have the "ask" property on them, then the
user will be presented with a list of choices of possible encodings and
asked to choose one. This list is typically sorted first by level of
likelihood, and then within this, by the order in which the subtypes
appear in the priority list. This list is displayed in a special kind
of dialog box or other buffer allowing the user, in addition to just
choosing a particular encoding, to view what the file would look like
if it were decoded according to the type.

   Furthermore, whenever a file is decoded according to a particular
type, the decoding engine keeps track of status values that are output
by the coding system type's decoding method. Generally, this status
will be in the form of errors or warnings of various levels, some of
which may be severe enough to stop the decoding entirely, and some of
which may either indicate definitely malformed data but from which it's
possible to recover, or simply data that appears rather questionable.
If any of these status values are reported during decoding, the user
will be informed of this and asked "are you sure?"  As part of the "are
you sure" dialog box or question, the user can display the results of
the decoding to make sure it's correct. If the user says "no, they're
not sure," then the same list of choices as previously mentioned will
be presented.

RFC: Autodetection
------------------

Also appeared under heading "Implementation of Coding System Priority
Lists in Various Locales" ?

   Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   Date: 11/1/1999 2:48 AM

     >>>>> "Hrvoje" == Hrvoje Niksic <hniksic@srce.hr> writes:

         [Ben sez:]

         >> You are perfectly free to set up your XEmacs like this, but
         >> XEmacs/Mule *will* autodetect by default if there is no
         >> Content-Type: info and no reason to believe we are dealing with
         >> binary files.

         Hrvoje> In that case, it will be a serious mistake to make
         Hrvoje> --with-mule the default, ever.  I think more care should
         Hrvoje> be shown in meeting the need of European users.

   Hrvoje, I don't understand what you are worrying about.  I suspect
you are worrying about Handa's hyperactive and obstinate Mule, not what
Ben has in mind.  Yes, Ben has said "better guessing," but that's
simply not reasonable without substantial language environment
information.  I think trying to detect Latin-1 vs Latin-2 in the POSIX
locale would be a big mistake, I think trying to guess Big 5 v. Shift
JIS in a European locale would be a big mistake.

   If Ben doesn't mean "more appropriate use of language environment
information" when he writes "better guessing," I, as much as you, want
to see how he plans to do that.  Ben?  ("Yes/no/oops I need to think
about it" is good enough if you have specifics you intend to put in the
RFC you're planning to present.)

   Let me give a formal proposal of what I would like to see in the
autodetection specification.

  1. Definitions

       1. "Autodetection" means detecting and making available to Mule
          the external file's encoding.  See (5), below.  It doesn't
          imply any specific actions based on that information.

       2. The "default" case is POSIX locale, and no environment
          information in ~/.xemacs/init.el.

          N.B.  This *will* cause breakage for all 1-byte users because
          the default case can no longer assume Latin-1.  You *may* be
          able to use the TTY font or the Xt -font option to fake this,
          and default to iso8859-1; I would hope that we would not use
          such a kludge in the beta versions, although it might be
          satisfactory for general use.  In particular, encodings like
          VISCII (Vietnamese) and I believe KOI-8 (Cyrillic) are not
          ISO-2022-clean, but using C1 control characters as a heuristic
          for detecting binary files is useful.

          If we do allow it, I think that XEmacs should bitch and warn
          that the practices of implicitly specifying language
          environment by -font and defaulting on TTYs is deprecated and
          likely to be obsoleted.

       3. The "European" case is any Latin-* locale, either implied by
          setlocale() and friends or set in ~/.xemacs/init.el.  Latin-1
          is specifically not given precedence over other Latin-*, or
          non-Latin or non-ISO-8859 for that matter.  I suspect but am
          not sure that this case extends to all ISO-8859 encodings, and
          possibly to non-ISO-8859 single-byte encodings like KOI-8r (in
          particular when combined in a class with ISO-8859 encodings).

       4. The "CJK" case is any CJK locale.  Japanese is specifically
          not given precedence over other Asian locales.

       5. For completeness, define the "Unicode" case (Unicode
          unfortunately has lots of junk such as precomposed characters,
          language tags, and directionality indicators in it; we
          probably don't care yet, but we should also not claim
          compliance) and the "general" case (which has a lot of
          features similar to Unicode, but lacks the advantage of a
          unified encoding).  This proposal has no idea how to handle
          the special features of these, or even if that matters.  The
          general case includes stuff that nobody here really knows how
          it works, like Tibetan and Ethiopic.

     Each of the following cases is given in the order of priority of
     detection.  I'm not sure I'm serious about the top priority given
     the (optional) Unicode detection.  This may be appropriate if Ben
     is right that ISO-2022 is going to disappear, but possibly not
     until then (two two-byte sequences out of 65536 is probably 1.99
     too many).  It probably isn't too risky if (6)(c) is taken pretty
     seriously; a Unicode file should contain _no_ private use
     characters unless the encoding is explicitly specified, and that's
     a block of 1/10 of the code space, which should help a lot in
     detecting binary files.

  2. Default locale

       1. Some Unicode (fixed width; maybe UTF-8, too?) may optionally
          be detected by the byte-order-mark magic (if the first two
          bytes are 0xFE 0xFF, the file is Unicode text, if 0xFF 0xFE,
          it is wrong-endian Unicode; if legal in UTF-8, it would be
          0xFE 0xBB 0xBF, either-endian).  This is probably an
          optimization that should not be on by default yet.

       2. ISO-2022 encodings will be detected as long as they use
          explicit designation of all non-ASCII character sets.  This
          means that many 7-bit ISO-2022 encodings would be detected
          (eg, ISO-2022-JP), but EUC-JP and X Compound Text would not,
          because they implicitly designate character sets.

          N.B. Latin-1 will be detected as binary, as for any Latin-*.

          N.B. An explicit ISO-2022 designation is semantically
          equivalent to a Content-Type: header.  It is more dangerous
          because shorter, but I think we should recognize them by
          default despite the slight risk; XEmacs is a text editor.

          N.B. This is unlikely to be as dangerous as it looks at first
          glance.  Any file that includes an 8-bit-set byte before the
          first valid designation should be detected as binary.

       3. Binary files will be detected (eg, presence of NULs, other
          non-whitespace control characters, absurdly long lines, and
          presence of bytes >127).

       4. Everything else is ASCII.

       5. Newlines will be detected in text files.

  3. European locales

       1. Unicode may optionally be detected by the byte-order-mark
          magic.

       2. ISO-2022 encodings will be detected as long as they use
          explicit designation of all non-ASCII character sets.

       3. A locale-specific class of 1-byte character sets (eg,
          '(Latin-1)) will be detected.

          N.B.  The reason for permitting a class is for cases like
          Cyrillic where there are both ISO-8859 encodings and
          incompatible encodings (KOI-8r) in common use.  If you want to
          write a Latin-1 v. Latin-2 detector, be my guest, but I don't
          think it would be easy or accurate.

       4. Binary files will be detected per (2)(c), except that only
          8-bit bytes out of the encoding's range imply binary.

       5. Everything else is ASCII.

       6. Newlines will be detected in text files.

  4. CJK locales

       1. Unicode may optionally be detected by the byte-order-mark
          magic.

       2. ISO-2022 encodings will be detected as long as they use
          explicit designation of all non-ASCII character sets.

       3. A locale-specific class of multi-byte and wide-character
          encodings will be detected.  N.B. No 1-byte character sets
          (eg, Latin-1) will be detected.  The reason for a class is to
          allow the Japanese to let Mule do the work of choosing EUC v.
          SJIS.

       4. Binary files will be detected per (3)(d).

       5. Everything else is ASCII.

       6. Newlines will be detected in text files.

  5. Unicode and general locales; multilingual use

       1. Hopefully a system general enough to handle (2)-(4) will
          handle these, too, but we should watch out for gotchas like
          Unicode "plane 14" tags which (I think _both_ Ben and Olivier
          will agree) have no place in the internal representation, and
          thus must be treated as out-of-band control sequences.  I
          don't know if all such gotchas will be as easy to dispose of.

       2. An explicit coding system priority list will be provided to
          allow multilingual users to autodetect both Shift JIS and Big
          5, say, but this ability is not promised by Mule, since it
          would involve (eg) heuristics like picking a set of code
          points that are frequent in Shift JIS and uncommon in Big 5
          and betting that a file containing many characters from that
          set is Shift JIS.

  6. Relationship to decoding semantics

       1. Autodetection should be run on every input stream unless the
          user explicitly disables it.

       2. The (conceptual) default procedure is

       3. Read the file into the buffer

          Announce the result of autodetection to the user.

          User may request decoding, with autodetected encoding(s)
          given priority in a list of available encodings.

          zations (see (e) below) should avoid introducing data tion
          that this default procedure would avoid.

          sly, it can't be perfect if any autodecoding is done; like
          Hrvoje should have an easily available option to  to this
          default (or an optimized approximation which t actually read
          the whole file into a buffer) or simply y everything as
          binary (with the "font" for binary files a user option).

       4. This implies that we should be detecting conditions in the
          tail of the file which violate the implicit assumptions of the
          coding system autodetected (eg, in UTF-8 illegal UTF-8
          sequences, including those corresponding to surrogates) should
          raise a warning; the buffer should probably be made read-only
          and the user prompted.

          This could be taken to extremes, like checking by table
          whether all characters in a Japanese file are actually
          legitimate JIS codes; that's insane (and would cause corporate
          encodings to be recognized as binary).  But we should think
          about the idea that autodetection shouldn't mean XEmacs can't
          change its mind.

       5. A flexible means for the user to delegate the decision
          (conditional on the result of autodetection) to decode or not
          to XEmacs or a Lisp program should be provided (eg, the
          coding priority list and/or a file-coding-alist).

       6. Optimized operations (eg, the current lstreams) should be
          provided, with the recognition that if they depend on sampling
          the file they are risky.

       7. Mule should provide a reasonable set of default delegations
          (as in (d) above) for as many locales as possible.

  7. Implementation

       1. I think all the decision logic suggested above can be
          accomplished through a coding-priority-list and appropriate
          initializations for different language environments, and a
          file-coding-alist.

       2. Many of the tests on the file's tail shouldn't be very
          expensive; in particular, all of the ones I've suggested are
          O(n) although they might involve moderate-sized auxiliary
          tables for efficiency (eg, 64kB for a single Unicode-oriented
          test).

   Other comments:

   It might be reasonable given Hrvoje's objections to require that any
autodetection that could cause data loss (any coding system that
involves escape sequences, and only those AFAIK: by design translation
to Unicode is invertible) by default prompt the user (presumable with a
novice-like ability to retain the prompt, always default to binary, or
always default to the autodetected encoding) in the future, at least in
locales that don't need it (POSIX, Latin-any).

   Ben thinks that we can remember the input data; I think it's going to
be hard to comprehensively test that a highly optimized version works.
Good design will help, but ISO-2022 is enormously complex, and there
are many encodings that violate even its lax assumptions.  On the other
hand, memory is the only way to get non-rewindable streams right.

   Hrvoje himself said he would like to have an XEmacs that
distinguishes between Latin-1 and Latin-2 text.  Where it is possible
to do that, this is exactly what autodetection of ISO-2022 and Unicode
gives you.  Many people would want that, even at some risk of binary
corruption.

   >> Once again I remind you that XEmacs is a *text* editor.  There
>> are lots of files that potentially may have Japanese etc. in     >>
them without this marked, e.g. C or Elisp files in the XEmacs     >>
source.  Surely you're not arguing that we interpret even these     >>
files as binary by default?

   Hrvoje> I am.  If I want to see Japanese, I'll setup my     Hrvoje>
environment that way.  But I don't, and neither do 99% of     Hrvoje>
Croatian users.  I can't speak for French, Italian, and     Hrvoje>
others, but I'd assume similar.

   Hrvoje> If there is Japanese in the source files, I will see it as
 Hrvoje> escape sequences, which is perfectly fine, because I don't
Hrvoje> read Japanese.

   And some (European) people will have their terminals scrambled,
because Shift-JIS contains sequences that can change the state of XTerm
(as do fixed-width Unicode and Big5).  This may also be a problem with
some Windows-12xx encodings; I'm not sure they all are ISO-2022-clean.
(This isn't a problem for XEmacs native X11 frames or native MS-Windows
frames, and the XEmacs sources themselves are all in 7-bit ISO-2022 now
IIRC.  But it is a potential source of great frustration for many
users.)

   I think that should be considered too, although it is presumably
lower priority than the data corruption of binary files.

Response to RFC: Autodetection
------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Date: 11/1/1999 7:24 AM

   Stephen, thank you very much for writing this up.  I think it is a
good start, and definitely moving in the direction I would like to see
things going: more proposals, less arguing. (aka "more light, less
heat") However, I have some suggestions for cleaning this up:

   You should try to make it more layered.  For example, you might have
one section devoted to the workings of autodetection, which starts out
like this (the section numbers below are totally arbitrary):

Section 5
.........

`Autodetect()' is a function whose arguments are (1) a readable stream,
(2) some hints indicating how the autodetection is to proceed, and (3)
a value indicating the maximum number of characters to examine at the
beginning of the stream.  (Possibly, the value in (3) may be some
special symbol indicating that we only go as far as the next line, or a
certain number of lines ahead; this would be used as part of
"continuous autodetection", e.g. we are decoding the results of an
interactive terminal session, where the user may periodically switch
encodings, line terminations, etc. as different programs get run and/or
telnet or similar sessions are entered into and exited.) We assume the
stream is rewindable; if not, insert a "rewinding" stream in front of
the non-rewinding stream; this kind of stream automatically buffers the
data as necessary.  [You can use pseudo-code terminology here.  No need
for straight C or ELisp.]  [Then proceed to describe what the hints
look like - e.g. you could portray it as a property list or whatever.
The idea is that, for each locale, there is a corresponding hints value
that is used at least by default.  The hints structure also has to be
set up to allow for two or more competing hints specifications to be
merged together.  For example, the extension of a file might provide an
additional hint or hints about how to interpret the data of that file,
and the caller of `autodetect()', when calling `autodetect()' on such a
file, would need to have a way of gracefully merging the default hints
corresponding to the locale with the more specific hints provided by the
extension.  Furthermore, users like Hrvoje might well want to provide
their own hints to supplement and override parts of the generic hints -
e.g. "I don't ever want to see non-European encodings decoded; treat
them as binary instead".]  [Then describe algorithmically how the
autodetection works.  First, you could describe it more generally, i.e.
presenting an algorithmic overview, then you could discuss in detail
exactly how autodetection of a particular type of external encoding
works - e.g. "for iso2022, we first look for an escape character,
followed by a byte in this range [. ... .] etc."]

Section 6
.........

This section describes the concept of a locale in XEmacs, and how it is
derived from the user's environment.  A locale in XEmacs is a pair, a
country and a language, together determining the handling of
locale-specific areas of XEmacs.  All locale-specific areas in XEmacs
make use of this XEmacs locale, and do not attempt to derive the locale
from any other sources.  The user is free to change the current locale
at any time; accessor and mutator functions are provided to do this so
that various locale-specific areas can optionally be changed together
with it.

   [Then you describe how the XEmacs locale is extracted from
`init.el', from `setlocale()', from the LANG environment variables,
from -font, or wherever else.  All other sections assume this dirty
work is done and never even mention it]

Section 7
.........

[Here you describe the default `autodetect()' hints value corresponding
to each possible locale.  You should probably use a schematic
description here, e.g.  an actual Lisp property list, liberally
commented.]

Section 8 etc.
..............

[Other sections cover anything I've missed.  By being very careful to
separate out the layers, you simultaneously introduce more rigor
(easier to catch bugs) and make it easier for someone else to
understand it completely.]

Better Algorithm, More Flexibility, Different Levels of Certainty
-----------------------------------------------------------------

Much More Flexible Coding System Priority List, per-Language Environment
------------------------------------------------------------------------

User Ability to Select Encoding when System Unsure or Encounters Errors
-----------------------------------------------------------------------

Another Autodetection Proposal
------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   however, in general the detection code has major problems and needs
lots of work:

   * instead of merely "yes" or "no" for particular categories, we need
     a more flexible system, with various levels of likelihood.
     Currently I've created a system with six levels, as follows:

     [see `file-coding.h']

     Let's consider what this might mean for an ASCII text detector.
     (In order to have accurate detection, especially given the
     iteration I proposed below, we need active detectors for *all*
     types of data we might reasonably encounter, such as ASCII text
     files, binary files, and possibly other sorts of ASCII files, and
     not assume that simply "falling back to no detection" will work at
     all well.)

     An ASCII text detector DOES NOT report ASCII text as level 0, since
     that's what the detector is looking for.  Such a detector ideally
     wants all bytes in the range 0x20 - 0x7E (no high bytes!), except
     for whitespace control chars and perhaps a few others; LF, CR, or
     CRLF sequences at regular intervals (where "regular" might mean an
     average < 100 chars and 99% < 300 for code and other stuff of the
     "text file w/line breaks" variety, but for the "text file w/o line
     breaks" variety, excluding blank lines, averages could easily be
     600 or more with 2000-3000 char "lines" not so uncommon); similar
     statistical variance between odds and evens (not Unicode);
     frequent occurrences of the space character; letters more common
     than non-letters; etc.  Also checking for too little variability
     between frequencies of characters and for exclusion of particular
     characters based on character ranges can catch ASCII encodings
     like base-64, UUEncode, UTF-7, etc.  Granted, this doesn't even
     apply to everything called "ASCII", and we could potentially
     distinguish off ASCII for code, ASCII for text, etc. as separate
     categories.  However, it does give us a lot to work off of, in
     deciding what likelihood to choose - and it shows there's in fact
     a lot of detectable patterns to look for even in something
     seemingly so generic as ASCII.  The detector would report most text
     files in level 1 or level 2.  EUC encodings, Shift-JIS, etc.
     probably go to level -1 because they also pass the EOL test and
     all other tests for the ASCII part of the text, but have lots of
     high bytes, which in essence turn them into binary.  Aberrant text
     files like something in BASE64 encoding might get placed in level
     0, because they pass most tests but fail dramatically the
     frequency test; but they should not be reported as any lower,
     because that would cause explicit prompting, and the user should
     be able any valid text file without prompting.  The escape
     sequences and the base-64-type checks might send 7-bit iso2022 to
     0, but probably not -1, for similar reasons.

   * The assumed algorithm for the above detection levels is to in
     essence sort categories first by detection level and then by
     priority.  Perhaps, however, we would want smarter algorithms, or
     at least something user-controllable - in particular, when (other
     than no category at level 0 or greater) do we prompt the user to
     pick a category?

   * Improvements in how the detection algorithm works: we want to
     handle lots of different ways something could be encoded,
     including multiple stacked encodings.  trying to specify a series
     of detection levels (check for base64 first, then check for gzip,
     then check for an i18n decoding, then for crlf) won't generally
     work.  for example, what about the same encoding appearing more
     than once? for example, take euc-jp, base64'd, then gzip'd, then
     base64'd again: this could well happen, and you could specify the
     encodings specifically as base64|gzip|base64|euc-jp, but we'd like
     to autodetect it without worrying about exactly what order these
     things appear in.  we should allow for iterating over
     detection/decoding cycles until we reach some maximum (we got
     stuck in a loop, due to incorrect category tables or detection
     algorithms), have no reported detection levels over -1, or we end
     up with no change after a decoding pass (i.e. the coding system
     associated with a chosen category was `no-conversion' or something
     equivalent).  it might make sense to divide things into two phases
     (internal and external), where the internal phase has a separate
     category list and would probably mostly end up handling EOL
     detection; but the i think about it, the more i disagree.  with
     properly written detectors, and properly organized tables (in
     general, those decodings that are more "distinctive" and thus
     detectable with greater certainty go lower on the list), we
     shouldn't need two phases.  for example, let's say the example
     above was also in CRLF format.  The EOL detector (which really
     detects *plain text* with a particular EOL type) would return at
     most level 0 for all results until the text file is reached,
     whereas the base64, gzip or euc-jp decoders will return higher.
     Once the text file is reached, the EOL detector will return 0 or
     higher for the CRLF encoding, and all other detectors will return
     0 or lower; thus, we will successfully proceed through CRLF
     decoding, or at worst prompt the user. (The only
     external-vs-internal distinction that might make sense here is to
     favor coding systems of the correct source type over those that
     require conversion between external and internal; if done right,
     this could allow the CRLF detector to return level 1 for all
     CRLF-encoded text files, even those that look like Base-64 or
     similar encoding, so that CRLF encoding will always get decoded
     without prompting, but not interfere with other decoders.  On the
     other hand, this external-vs-internal distinction may not matter
     at all - with automatic internal-external conversion, CRLF
     decoding can occur before or after decoding of euc-jp, base64,
     iso2022, or similar, without any difference in the final results.)

     #### What are we trying to say?  In base64, the CRLF decoding
     before base64 decoding is irrelevant, they will be thrown out as
     whitespace is not significant in base64.

     [sjt considers all of this to be rather bogus.  Ideas like "greater
     certainty" and "distinctive" can and should be quantified.  The
     issue of proper table organization should be a question of
     optimization.]

     [sjt wonders if it might not be a good idea to use Unicode's
     newline character as the internal representation so that (for
     non-Unicode coding systems) we can catch EOL bugs on Unix too.]

   * There need to be two priority lists and two
     category->coding-system lists.  Once is general, the other
     category->langenv-specific.  The user sets the former, the langenv
     category->the latter.  The langenv-specific entries take precedence
     category->over the others.  This works similarly to the
     category->category->Unicode charset priority list.

   * The simple list of coding categories per detectors is not enough.
     Instead of coding categories, we need parameters.  For example,
     Unicode might have separate detectors for UTF-8, UTF-7, UTF-16,
     and perhaps UCS-4; or UTF-16/UCS-4 would be one detection type.
     UTF-16 would have parameters such as "little-endian" and "needs
     BOM", and possibly another one like "collapse/expand/leave alone
     composite sequences" once we add this support.  Usually these
     parameters correspond directly to a coding system parameter.
     Different likelihood values can be specified for each parameter as
     well as for the detection type as a whole.  The user can specify
     particular coding systems for a particular combination of
     detection type and parameters, or can give "default parameters"
     associated with a detection type.  In the latter case, we create a
     new coding system as necessary that corresponds to the detected
     type and parameters.

   * a better means of presentation.  rather than just coming up with
     the new file decoded according to the detected coding system,
     allow the user to browse through the file and conveniently reject
     it if it looks wrong; then detection starts again, but with that
     possibility removed.  in cases where certainty is low and thus
     more than one possibility is presented, the user can browse each
     one and select one or reject them all.

   * fail-safe: even after the user has made a choice, if they later on
     realize they have the wrong coding system, they can go back, and
     we've squirreled away the original data so they can start the
     process over.  this may be tricky.

   * using a larger buffer for detection.  we use just a small piece,
     which can give quite random results.  we may need to buffer up all
     the data we look through because we can't necessarily rewind.  the
     idea is we proceed until we get a result that's at least at a
     certain level of certainty (e.g. "probable") or we reached a
     maximum limit of how much we want to buffer.

   * dealing with interactive systems.  we might need to go ahead and
     present the data before we've finished detection, and then
     re-decode it, perhaps multiple times, as we get better detection
     results.

   * Clearly some of these are more important than others.  at the very
     least, the "better means of presentation" should be implemented as
     soon as possible, along with a very simple means of fail-safe
     whenever the data is readibly available, e.g. it's coming from a
     file, which is the most common scenario.

   ben [at least that's what sjt thinks]

   *****

   Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   While this is clearly something of an improvement over earlier
designs, it doesn't deal with the most important issue: to do better
than categories (which in the medium term is mostly going to mean
"which flavor of Unicode is this?"), we need to look at statistical
behavior rather than ruling out categories via presence of specific
sequences.  This means the stream processor should

  1. keep octet distributions (octet, 2-, 3-, 4- octet sequences)

  2. in some kind of compressed form

  3. look for "skip features" (eg, characteristic behavior of leading
     bytes for UTF-7, UTF-8, UTF-16, Mule code)

  4. pick up certain "simple" regexps

  5. provide "triggers" to determine when statistical detectors should
     be invoked, such as octet count

  6. and "magic" like Unicode signatures or file(1) magic.

   -sjt


File: internals.info,  Node: Future Work -- Conversion Error Detection,  Next: Future Work -- Unicode,  Prev: Future Work -- Autodetection,  Up: Future Work -- Byte Code Snippets

44.15.2 Future Work - Conversion Error Detection
------------------------------------------------

"No Corruption" Scheme for Preserving External Encoding when Non-Invertible Transformation Applied
--------------------------------------------------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   A preliminary and simple implementation is:

     But you could implement it much more simply and usefully by just
     determining, for any text being decoded into mule-internal, can we
     go back and read the source again?  If not, remember the entire
     file (GNUS message, etc) in text properties.  Then, implement the
     UI interface (like Netscape's) on top of that.  This way, you have
     something that at least works, but it might be inefficient.  All we
     would need to do is work on making the underlying implementation
     more efficient.

   A more detailed proposal for avoiding binary file corruption is

     Basic idea: A coding system is a filter converting an entire input
     stream into an output stream. The resulting stream can be said to
     be "correspondent to" the input stream. Similarly, smaller units
     can correspond. These could potentially include zero width
     intervals on either side, but we avoid this.  Specifically, the
     coding system works like:

          loop (input) {

           Read bytes till we have enough to generate a translated character or a chars.

           This establishes a "correspondence" between the whole input and
           output more or less in minimal chunks.

          }

     We then do the following processing:

       1. Eliminate correspondences where one or the other of the I/O
          streams has a zero interval by combining with an adjacent
          interval;

       2. Group together all adjacent "identity" correspondences into as
          large groups as possible;

       3. Use text properties to store the non-identity correspondences
          on the characters. For identity correspondences, use a simple
          text property on all that contains no data but just indicates
          that the whole string of text is identity corresponded. (How
          do we define "identity"? Latin 1 or could it be something
          else? For example, Latin 2)?

       4. Figure out the procedures when text is inserted/deleted and
          copied or pasted.

       5. Figure out to save the file out making use of the
          correspondences. Allow ways of saving without
          correspondences, and doing a "save to buffer with and without
          correspondences."  Need to be clever when dealing with modal
          coding systems to parse the correspondences to get the
          internal state right.

Another Error-Catching Idea
---------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Nov 4, 1999

   Finally, I don't think "save the input" is as hard as you make it
out to be.  Conceptually, in fact, it's simple: for each minimal group
of bytes where you cannot absolutely guarantee that an
external->internal transformation is reversible, you put a text
property on the corresponding internal character indicating the bytes
that generated this character.  We also put a text property on every
character, indicating the coding system that caused the transformation.
This latter text property is extremely efficient (e.g. in a buffer
with no data pasted from elsewhere, it will map to a single extent over
all the buffer), and the former cases should not be prevalent enough to
cause a lot of inefficiency, esp. if we define what "reversible" means
for each coding system in such a way that it correctly handles the most
common cases.  The hardest part, in fact, is making all the string/text
handling in XEmacs be robust w.r.t. text properties.

Strategies for Error Annotation and Coding Orthogonalization
------------------------------------------------------------

Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   We really want to separate out a number of things.  Conceptually,
there is a nested syntax.

   At the top level is the ISO 2022 extension syntax, including charset
designation and invocation, and certain auxiliary controls such as the
ISO 6429 direction specification.  These are octet-oriented, with the
single exception (AFAIK) of the "exit Unicode" sequence which uses the
UTF's natural width (1 byte for UTF-7 and UTF-8, 2 bytes for UCS-2 and
UTF-16, and 4 bytes for UCS-4 and UTF-32).  This will be treated as a
(deprecated) special case in Unicode processing.

   The middle layer is ISO 2022 character interpretation.  This will
depend on the current state of the ISO 2022 registers, and assembles
octets into the character's internal representation.

   The lowest level is translating system control conventions.  At
present this is restricted to newline translation, but one could
imagine doing tab conversion or line wrapping here.  "Escape from
Unicode" processing would be done at this level.

   At each level the parser will verify the syntax.  In the case of a
syntax error or warning (such as a redundant escape sequence that
affects no characters), the parser will take some action, typically
inserting the erroneous octets directly into the output and creating an
annotation which can be used by higher level I/O to mark the affected
region.

   This should make it possible to do something sensible about
separating newline convention processing from character construction,
and about preventing ISO 2022 escape sequences from being recognized
inappropriately.

   The basic strategy will be to have octet classification tables, and
switch processing according to the table entry.

   It's possible that, by doing the processing with tables of functions
or the like, the parser can be used for both detection and translation.

Handling Writing a File Safely, Without Data Loss
-------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

     When writing a file, we need error detection; otherwise somebody
     will create a Unicode file without realizing the coding system of
     the buffer is Raw, and then lose all the non-ASCII/Latin-1 text
     when it's written out.  We need two levels

       1. first, a "safe-charset" level that checks before any actual
          encoding to see if all characters in the document can safely
          be represented using the given coding system.  FSF has a
          "safe-charset" property of coding systems, but it's stupid
          because this information can be automatically derived from
          the coding system, at least the vast majority of the time.
          What we need is some sort of
          alternative-coding-system-precedence-list, langenv-specific,
          where everything on it can be checked for safe charsets and
          then the user given a list of possibilities.  When the user
          does "save with specified encoding", they should see the same
          precedence list.  Again like with other precedence lists,
          there's also a global one, and presumably all coding systems
          not on other list get appended to the end (and perhaps not
          checked at all when doing safe-checking?).  safe-checking
          should work something like this: compile a list of all
          charsets used in the buffer, along with a count of chars
          used.  that way, "slightly unsafe" coding systems can perhaps
          be presented at the end, which will lose only a few characters
          and are perhaps what the users were looking for.

          [sjt sez this whole step is a crock.  If a universal coding
          system is unacceptable, the user had better know what he/she
          is doing, and explicitly specify a lossy encoding.  In
          principle, we can simply check for characters being writable
          as we go along.  Eg, via an "unrepresentable character
          handler."  We still have the buffer contents.  If we can't
          successfully save, then ask the user what to do.  (Do we ever
          simply destroy previous file version before completing a
          write?)]

       2. when actually writing out, we need error checking in case an
          individual char in a charset can't be written even though the
          charsets are safe.  again, the user gets the choice of other
          reasonable coding systems.

          [sjt - something is very confused, here; safe charsets should
          be defined as those charsets all of whose characters can be
          encoded.]

       3. same thing (error checking, list of alternatives, etc.) needs
          to happen when reading!  all of this will be a lot of work!

   Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   I don't much like Ben's scheme.  First, this isn't an issue of I/O,
it's a coding issue.  It can happen in many places, not just on stream
I/O.  Error checking should take place on all translations.  Second,
the two-pass algorithm should be avoided if possible.  In some cases
(eg, output to a tty) we won't be able to go back and change the
previously output data.  Third, the whole idea of having a buffer full
of arbitrary characters which we're going to somehow shoehorn into a
file based on some twit user's less than informed idea of a coding
system is kind of laughable from the start.  If we're going to say that
a buffer has a coding system, shouldn't we enforce restrictions on what
you can put into it?  Fourth, what's the point of having safe charsets
if some of the characters in them are unsafe?  Fifth, what makes you
think we're going to have a list of charsets?  It seems to me that
there might be reasons to have user-defined charsets (eg, "German" vs
"French" subsets of ISO 8859/15).  Sixth, the idea of having language
environment determine precedence doesn't seem very useful to me.  Users
who are working with a language that corresponds to the language
environment are not going to run into safe charsets problems.  It's
users who are outside of their usual language environment who run into
trouble.  Also, the reason for specifying anything other than a
universal coding system is normally restrictions imposed by other users
or applications.  Seventh, the statistical feedback isn't terribly
useful.  Users rarely "want" a coding system, they want their file
saved in a useful way.  We could add a FORCE argument to conversions
for those who really want a specific coding system.  But mostly, a user
might want to edit out a few unsafe characters.  So (up to some
maximum) we should keep a list of unsafe text positions, and provide a
convenient function for traversing them.

   -sjt


File: internals.info,  Node: Future Work -- Unicode,  Next: Future Work -- BIDI Support,  Prev: Future Work -- Conversion Error Detection,  Up: Future Work -- Byte Code Snippets

44.15.3 Future Work - Unicode
-----------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Following is an old proposal.  Unicode has been implemented already,
in a different fashion; but there are some ideas here for more general
support, e.g. properties of Unicode characters other than their mappings
to particular charsets.

   We recognize 128, [256], 128x128, [256x256] for source charsets;

   for Unicode, 256x256 or 16x256x256.

   In all cases, use tables of tables and substitute a default subtable
if entire row is empty.

   If destination is Unicode, either 16 or 32 bits.

   If destination is charset, either 8 or 16 bits.

   For the moment, since we only do 94, 96, 94x94 or 96x96, only do 128
or 128x128 for source charsets and use the range 33-126 or 32-127.
(Except ASCII - we special case that and have no table because we can
algorithmically translate)

   Also have a 16x256x256 table -> 32 bits of Unicode char properties.

   A particular charset contains two associated mapping tables, for both
directions.

   API is set-unicode-mapping:

     (set-unicode-mapping
          unicode char
          unicode charset-code charset-offset
          unicode vector of char
          unicode list of char
          unicode string of char
          unicode vector or list of codes charset-offset

   Establishes a mapping between a unicode codepoint (a fixnum) and
one or more chars in a charset.  The mapping is automatically
established in both directions.  Chars in a charset can be specified
either with an actual character or a codepoint (i.e. an fixnum)   and
the charset it's within.  If a sequence of chars or charset   points is
given, multiple mappings are established for consecutive   unicode
codepoints starting with the given one.  Charset codepoints   are
specified as most-significant x 256 + least significant, with   both
bytes in the range 33-126 (for 94 or 94x94) or 32-127 (for 96   or
96x96), unless an offset is given, which will be subtracted from   each
byte.  (Most common values are 128, for codepoints given with   the
high bit set, or -32, for codepoints given as 1-94 or 0-95.)

   Other APIs:

     (write-unicode-mapping file charset)

   Write the mapping table for a particular charset to the specified
file.  The tables are written in an internal format that allows for
efficient loading, for portability across platforms and XEmacs
invocations, for conserving space, for appending multiple tables one
directly after another with no need for a directory anywhere in the
file, and for reorganizing a file as in this format (with a magic
sequence at the beginning).  The data will be appended at the end of
a file, so that multiple tables can be written to a file; remove the
file first to avoid this.

     (write-unicode-properties file unicode-codepoint length)

   Write the Unicode properties (not including charset mappings) for
the specified range of contiguous Unicode codepoints to the end of
the file (i.e. append mode) in a binary format similar to what was
mentioned in the write-unicode-mapping description and with the same
features.

   Extension to set-unicode-mapping:

     (set-unicode-mapping
       list-or-vector-of-unicode-codepoints char
       ""                                   charset-code charset-offset
       ""                                   sequence of char
       ""                                   list-or-vector-of-codes
     charset-offset

   The first two forms are conceptually the inverse of the forms above
to specify characters for a contiguous range of Unicode codepoints.
These new forms let you specify the Unicode codepoints for a
contiguous range of chars in a charset.  "Contiguous" here means   that
if we run off the end of a row, we go to the first entry of the   next
row, rather than to an invalid code point.  For example, in a   94x94
charset, valid rows and columns are in the range 0x21-0x7e;   after
0x457c 0x457d 4x457e goes 0x4621, not something like 0x457f,   which is
invalid.

   The final two forms are the most general, letting you specify an
arbitrary set of both Unicode points and charset chars, and the two
are matched up just like a series of individual calls.  However, if
the lists or vectors do not have the same length, an error is
signaled.

     (load-unicode-mapping file &optional charset)

   If charset is omitted, loads all charset mapping tables found and
returns a list of the charsets found.  If charset is specified,
searches through the file for the appropriate mapping tables.  (This
is extremely fast because each entry in the file gives an offset to
the next one).  Returns t if found.

     (load-unicode-properties file unicode-codepoint)

     (list-unicode-entries file)

     (autoload-unicode-mapping charset)

   ...

   (unfinished)


File: internals.info,  Node: Future Work -- BIDI Support,  Next: Future Work -- Localized Text/Messages,  Prev: Future Work -- Unicode,  Up: Future Work -- Byte Code Snippets

44.15.4 Future Work - BIDI Support
----------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

  1. Use text properties to handle nesting levels, overrides
     BIDI-specific text properties (as per Unicode BIDI algorithm)
     computed at text insertion time.

  2. Lisp API for reordering a display line at redisplay time, possibly
     substitution of different glyphs (esp. mirroring of glyphs).

  3. Lisp API called after a display line is laid out, but only when
     reordering may be necessary (display engine checks for non-uniform
     BIDI text properties; can handle internally a line that's
     completely in one direction)

  4. Default direction is a buffer-local variable

  5. We concentrate on implementing Unicode BIDI algorithm.

  6. Display support for mirroring of entire window

  7. Display code keeps track of mirroring junctures so it can display
     double cursor.

  8. Entire layout of screen (on a per window basis) is exported as a
     Lisp API, for visual editing (also very useful for other purposes
     e.g. proper handling of word wrapping with proportional fonts,
     complex Lisp layout engines e.g. W3)

  9. Logical, visual, etc. cursor movement handled entirely in Lisp,
     using aforementioned API, plus a specifier for controlling how
     cursor is shown (e.g. split or not).


File: internals.info,  Node: Future Work -- Localized Text/Messages,  Prev: Future Work -- BIDI Support,  Up: Future Work -- Byte Code Snippets

44.15.5 Future Work - Localized Text/Messages
---------------------------------------------

NOTE: There is existing message translation in X Windows of menu names.
This is handled through X resources.  The files are in
`PACKAGES/mule-packages/locale/app-defaults/LOCALE/Emacs', where LOCALE
is `ja', `fr', etc.

   See lib-src/make-msgfile.lex.

   Long comment from jwz, some additions from ben marked "ben":

   (much of this comment is outdated, and a lot of it is actually
implemented)

44.15.6 Proposal for How This All Ought to Work
-----------------------------------------------

Author: Jamie Zawinski (mailto:jwz@jwz.org)

   this isn't implemented yet, but this is the plan-in-progress

   In general, it's accepted that the best way to internationalize is
for all messages to be referred to by a symbolic name (or number) and
come out of a table or tables, which are easy to change.

   However, with Emacs, we've got the task of internationalizing a huge
body of existing code, which already contains messages internally.

   For the C code we've got two options:

   * Use a Sun-like `gettext()' form, which takes an "english" string
     which appears literally in the source, and uses that as a hash key
     to find a translated string;

   * Rip all of the strings out and put them in a table.

   In this case, it's desirable to make as few changes as possible to
the C code, to make it easier to merge the code with the FSF version of
emacs which won't ever have these changes made to it.  So we should go
with the former option.

   The way it has been done (between 19.8 and 19.9) was to use
`gettext()', but *also* to make massive changes to the source code.
The goal now is to use `gettext()' at run-time and yet not require a
textual change to every line in the C code which contains a string
constant.  A possible way to do this is described below.

   (`gettext()' can be implemented in terms of `catgets()' for non-Sun
systems, so that in itself isn't a problem.)

   For the Lisp code, we've got basically the same options: put
everything in a table, or translate things implicitly.

   Another kink that lisp code introduces is that there are thousands
of third- party packages, so changing the source for all of those is
simply not an option.

   Is it a goal that if some third party package displays a message
which is one we know how to translate, then we translate it?  I think
this is a worthy goal.  It remains to be seen how well it will work in
practice.

   So, we should endeavor to minimize the impact on the lisp code.
Certain primitive lisp routines (the stuff in lisp/prim/, and
especially in `cmdloop.el' and `minibuf.el') may need to be changed to
know about translation, but that's an ideologically clean thing to do
because those are considered a part of the emacs substrate.

   However, if we find ourselves wanting to make changes to, say,
RMAIL, then something has gone wrong.  (Except to do things like remove
assumptions about the order of words within a sentence, or how
pluralization works.)

   There are two parts to the task of displaying translated strings to
the user: the first is to extract the strings which need to be
translated from the sources; and the second is to make some call which
will translate those strings before they are presented to the user.

   The old way was to use the same form to do both, that is,
`GETTEXT()' was both the tag that we searched for to build a catalog,
and was the form which did the translation.  The new plan is to
separate these two things more: the tags that we search for to build
the catalog will be stuff that was in there already, and the
translation will get done in some more centralized, lower level place.

   This program (`make-msgfile.c') addresses the first part, extracting
the strings.

   For the emacs C code, we need to recognize the following patterns:

       message ("string" ... )
       error ("string")
       report_file_error ("string" ... )
       signal_simple_error ("string" ... )
       signal_simple_error_2 ("string" ... )

       build_translated_string ("string")
       #### add this and use it instead of `build_cistring()' in some places.

       yes_or_no_p ("string" ... )
       #### add this instead of funcalling Qyes_or_no_p directly.

       barf_or_query_if_file_exists	#### restructure this
       check all callers of Fsignal	#### restructure these
       signal_error (Qerror ... )		#### change all of these to `error()'

       And we also parse out the `interactive' prompts from `DEFUN()' forms.

       #### When we've got a string which is a candidate for translation, we
       should ignore it if it contains only format directives, that is, if
       there are no alphabetic characters in it that are not a part of a `%'
       directive.  (Careful not to translate either "%s%s" or "%s: ".)

   For the emacs Lisp code, we need to recognize the following patterns:

       (message "string" ... )
       (error "string" ... )
       (format "string" ... )
       (read-from-minibuffer "string" ... )
       (read-shell-command "string" ... )
       (y-or-n-p "string" ... )
       (yes-or-no-p "string" ... )
       (read-file-name "string" ... )
       (temp-minibuffer-message "string")
       (query-replace-read-args "string" ... )

   I expect there will be a lot like the above; basically, any function
which is a commonly used wrapper around an eventual call to `message' or
`read-from-minibuffer' needs to be recognized by this program.

       (dgettext "domain-name" "string")		#### do we still need this?

       things that should probably be restructured:
         `princ' in `cmdloop.el'
         `insert' in `debug.el'
         face-interactive
         `help.el', `syntax.el' all messed up

   Author: Ben Wing (mailto:ben@xemacs.org)

   ben: (format) is a tricky case.  If I use format to create a string
that I then send to a file, I probably don't want the string translated.
On the other hand, If the string gets used as an argument to (y-or-n-p)
or some such function, I do want it translated, and it needs to be
translated before the %s and such are replaced.  The proper solution
here is for (format) and other functions that call gettext but don't
immediately output the string to the user to add the translated (and
formatted) string as a string property of the object, and have
functions that output potentially translated strings look for a
"translated string" property.  Of course, this will fail if someone
does something like

        (y-or-n-p (concat (if you-p "Do you " "Does he ")
          		(format "want to delete %s? " filename))))

   But you shouldn't be doing things like this anyway.

   ben: Also, to avoid excessive translating, strings should be marked
as translated once they get translated, and further calls to gettext
don't do any more translating.  Otherwise, a call like

        (y-or-n-p (format "Delete %s? " filename))

   would cause translation on both the pre-formatted and post-formatted
strings, which could lead to weird results in some cases (y-or-n-p has
to translate its argument because someone could pass a string to it
directly).  Note that the "translating too much" solution outlined
below could be implemented by just marking all strings that don't come
from a .el or .elc file as already translated.

   Menu descriptors: one way to extract the strings in menu labels
would be to teach this program about "^(defvar .*menu\n" forms; that's
probably kind of hard, though, so perhaps a better approach would be to
make this program recognize lines of the form

       "string" ... ;###translate

   where the magic token ";###translate" on a line means that the string
constant on this line should go into the message catalog.  This is
analogous to the magic ";###autoload" comments, and to the magic
comments used in the EPSF structuring conventions.

   ---- So this program manages to build up a catalog of strings to be
translated.  To address the second part of the problem, of actually
looking up the translations, there are hooks in a small number of low
level places in emacs.

   Assume the existence of a C function gettext(str) which returns the
translation of STR if there is one, otherwise returns STR.

   * `message()' takes a char* as its argument, and always filters it
     through `gettext()' before displaying it.

   * errors are printed by running the lisp function `display-error'
     which doesn't call `message' directly (it princ's to streams), so
     it must be carefully coded to translate its arguments.  This is
     only a few lines of code.

   * `Fread_minibuffer_internal()' is the lowest level interface to all
     minibuf interactions, so it is responsible for translating the
     value that will go into Vminibuf_prompt.

   * Fpopup_menu filters the menu titles through `gettext()'.

     The above take care of 99% of all messages the user ever sees.

   * The lisp function temp-minibuffer-message translates its arg.

   * query-replace-read-args is funny; it does   (setq from
     (read-from-minibuffer (format "%s: " string) ... ))   (setq to
     (read-from-minibuffer (format "%s %s with: " string from) ... ))

   What should we do about this?  We could hack query-replace-read-args
to translate its args, but might this be a more general problem?  I
don't think we ought to translate all calls to format.  We could just
change the calling sequence, since this is odd in that the first %s
wants to be translated but the second doesn't.

   Solving the "translating too much" problem:

   The concern has been raised that in this situation:

   * "Help" is a string for which we know a translation;

   * someone visits a file called Help, and someone does something
     contrived like (error buffer-file-name)

   then we would display the translation of Help, which would not be
correct.  We can solve this by adding a bit to Lisp_String objects
which identifies them as having been read as literal constants from a
.el or .elc file (as opposed to having been constructed at run time as
it would in the above case.)  To solve this:

   * `Fmessage()' takes a lisp string as its first argument.  If that
     string is a constant, that is, was read from a source file as a
     literal, then it calls `message()' with it, which translates.
     Otherwise, it calls `message_no_translate()', which does not
     translate.

   * `Ferror()' (actually, `Fsignal()' when condition is Qerror) works
     similarly.

   More specifically, we do:

      Scan specified C and Lisp files, extracting the following
     messages:

             C files:
                GETTEXT (...)
                DEFER_GETTEXT (...)
                DEFUN interactive prompts
             Lisp files:
                (gettext ...)
                (dgettext "domain-name" ...)
                (defer-gettext ...)
                (interactive ...)

     The arguments given to this program are all the C and Lisp source
     files of GNU Emacs.  .el and .c files are allowed.  There is no
     support for .elc files at this time, but they may be specified;
     the corresponding .el file will be used.  Similarly, .o files can
     also be specified, and the corresponding .c file will be used.
     This helps the makefile pass the correct list of files.

     The results, which go to standard output or to a file specified
     with -a or -o (-a to append, -o to start from nothing), are quoted
     strings wrapped in gettext(...).  The results can be passed to
     xgettext to produce a .po message file.

     However, we also need to do the following:

       1. Definition of Arg below won't handle a generalized argument
          as might appear in a function call.  This is fine for DEFUN
          and friends, because only simple arguments appear there; but
          it might run into problems if Arg is used for other sorts of
          functions.

       2. `snarf()' should be modified so that it doesn't output null
          strings and non-textual strings (see the comment at the top
          of `make-msgfile.c').

       3. parsing of (insert) should snarf all of the arguments.

       4. need to add set-keymap-prompt and deal with gettext of that.

       5. parsing of arguments should snarf all strings anywhere within
          the arguments, rather than just looking for a string as the
          argument.  This allows if statements as arguments to get
          parsed.

       6. `begin_paren_counting()' et al. should handle recursive entry.

       7. handle set-window-buffer and other such functions that take a
          buffer as the other-than-first argument.

       8. there is a fair amount of work to be done on the C code.
          Look through the code for #### comments associated with
          '#ifdef I18N3' or with an I18N3 nearby.

       9. Deal with `get-buffer-process' et al.

      10. Many of the changes in the Lisp code marked 'rewritten for
          I18N3 snarfing' should be undone once (5) is implemented.

      11. Go through the Lisp code in prim and make sure that all
          strings are gettexted as necessary.  This may reveal more
          things to implement.

      12. Do the equivalent of (8) for the Lisp code.

      13. Deal with parsing of menu specifications.


File: internals.info,  Node: Future Work -- Lisp Stream API,  Next: Future Work -- Multiple Values,  Prev: Future Work -- Byte Code Snippets,  Up: Future Work

44.16 Future Work - Lisp Stream API
===================================

Author: Ben Wing (mailto:ben@xemacs.org)

   Expose XEmacs internal lstreams to Lisp as stream objects.  (In
addition to the functions given below, each stream object has
properties that can be associated with it using the standard put, get
etc. API.  For GNU Emacs, where put and get have not been extended to
be general property functions, but work only on strings, we would have
to create functions set-stream-property, stream-property,
remove-stream-property, and stream-properties.  These provide the same
functionality as the generic get, put, remprop, and object-plist
functions under XEmacs)

   (Implement properties using a hash table, and *generalize* this so
that it is extremely easy to add a property interface onto any kind of
object)

     (write-stream STREAM STRING)

   Write the STRING to the STREAM.  This will signal an error if all the
bytes cannot be written.

     (read-stream STREAM &optional N SEQUENCE)

   Reads data from STREAM.  N specifies the number of bytes or
characters, depending on the stream.  SEQUENCE specifies where to write
the data into.  If N is not specified, data is read until end of file.
If SEQUENCE is not specified, the data is returned as a stream.  If
SEQUENCE is specified, the SEQUENCE must be large enough to hold the
data.

     (push-stream-marker STREAM)

   returns ID, probably a stream marker object

     (pop-stream-marker STREAM)

   backs up stream to last marker

     (unread-stream STREAM STRING)

   The only valid STREAM is an input stream in which case the data in
STRING is pushed back and will be read ahead of all other data.  In
general, there is no limit to the amount of data that can be unread or
the number of times that unread-stream can be called before another
read.

     (stream-available-chars STREAM)

   This returns the number of characters (or bytes) that can definitely
be read from the screen without an error.  This can be useful, for
example, when dealing with non-blocking streams when an attempt to read
too much data will result in a blocking error.

     (stream-seekable-p STREAM)

   Returns true if the stream is seekable.  If false, operations such as
seek-stream and stream-position will signal an error.  However, the
functions set-stream-marker and seek-stream-marker will still succeed
for an input stream.

     (stream-position STREAM)

   If STREAM is a seekable stream, returns a position which can be
passed to seek-stream.

     (seek-stream STREAM N)

   If STREAM is a seekable stream, move to the position indicated by N,
otherwise signal an error.

     (set-stream-marker STREAM)

   If STREAM is an input stream, create a marker at the current
position, which can later be moved back to.  The stream does not need
to be a seekable stream.  In this case, all successive data will be
buffered to simulate the effect of a seekable stream.  Therefore use
this function with care.

     (seek-stream-marker STREAM marker)

   Move the stream back to the position that was stored in the marker
object. (this is generally an opaque object of type stream-marker).

     (delete-stream-marker MARKER)

   Destroy the stream marker and if the stream is a non-seekable stream
and there are no other stream markers pointing to an earlier position,
frees up some buffering information.

     (delete-stream STREAM N)

     (delete-stream-marker STREAM ID)

     (close-stream stream)

   Writes any remaining data to the stream and closes it and the object
to which it's attached.  This also happens automatically when the
stream is garbage collected.

     (getchar-stream STREAM)

   Return a single character from the stream. (This may be a single byte
depending on the nature of the stream).  This is actually a macro with
an extremely efficient implementation (as efficient as you can get in
Emacs Lisp), so that this can be used without fear in a loop.  The
implementation works by reading a large amount of data into a vector
and then simply using the function AREF to read characters one by one
from the vector.  Because AREF is one of the primitives handled
specially by the byte interpreter, this will be very efficient.  The
actual implementation may in fact use the function
call-with-condition-handler to avoid the necessity of checking for
overflow.  Its typical implementation is to fetch the vector containing
the characters as a stream property, as well as the index into that
vector.  Then it retrieves the character and increments the value and
stores it back in the stream.  As a first implementation, we check to
see when we are reading the character whether the character would be
out of range.  If so, we read another 4096 characters, storing them
into the same vector, setting the index back to the beginning, and then
proceeding with the rest of the getchar algorithm.

     (putchar-stream STREAM CHAR)

   This is similar to getchar-stream but it writes data instead of
reading data.

     Function make-stream

   There are actually two stream-creation functions, which are:

     (make-input-stream TYPE PROPERTIES)
     (make-output-stream TYPE PROPERTIES)

   These can be used to create a stream that reads data, or writes data,
respectively.  PROPERTIES is a property list and the allowable
properties in it are defined by the type.  Possible types are:

  1. `file' (this reads data from a file or writes to a file)

     Allowable properties are:

    `:file-name'
          (the name of the file)

    `:create'
          (for output streams only, creates the file if it doesn't
          already exist)

    `:exclusive'
          (for output streams only, fails if the file already exists)

    `:append'
          (for output streams only; starts appending to the end of the
          file rather than overwriting the file)

    `:offset'
          (positions in bytes in the file where reading or writing
          should begin.  If unspecified, defaults to the beginning of
          the file or to the end of the file when :appended specified)

    `:count'
          (for input streams only, the number of bytes to read from the
          file before signaling "end of file".  If nil or omitted, the
          number of bytes is unlimited)

    `:non-blocking'
          (if true, reads or writes will fail if the operation would
          block.  This only makes sense for non-regular files).

  2. `process' (For output streams only, send data to a process.)

     Allowable properties are:

    `:process'
          (the process object)

  3. `buffer'  (Read from or write to a buffer.)

     Allowable properties are:

    `:buffer'
          (the name of the buffer or the buffer object.)

    `:start'
          (the position to start reading from or writing to.  If nil,
          use the buffer point.  If true, use the buffer's point and
          move point beyond the end of the data read or written.)

    `:end'
          (only for input streams, the position to stop reading at.  If
          nil, continue to the end of the buffer.)

    `:ignore-accessible'
          (if true, the default for :start and :end ignore any
          narrowing of the buffer.)

  4. `stream' (read from or write to a lisp stream)

     Allowable properties are:

    `:stream'
          (the stream object)

    `:offset'
          (the position to begin to be reading from or writing to)

    `:length'
          (For input streams only, the amount of data to read,
          defaulting to the rest of the data in the string.  Revise
          string for output streams only if true, the stream is resized
          as necessary to accommodate data written off the end,
          otherwise the writes will fail.

  5. `memory' (For output only, writes data to an internal memory
     buffer.  This is more lightweight than using a Lisp buffer.  The
     function memory-stream-string can be used to convert the memory
     into a string.)

  6. `debugging' (For output streams only, write data to the debugging
     output.)

  7. `stream-device' (During non-interactive invocations only, Read
     from or write to the initial stream terminal device.)

  8. `function' (For output streams only, send data by calling a
     function, exactly as with the STREAM argument to the print
     primitive.)

     Allowable Properties are:

    `:function'
          (the function to call.  The function is called with one
          argument, the stream.)

  9. `marker' (Write data to the location pointed to by a marker and
     move the marker past the data.)

     Allowable properties are:

    `:marker'
          (the marker object.)

 10. `decoding' (As an input stream, reads data from another stream and
     decodes it according to a coding system.  As an output stream
     decodes the data written to it according to a coding system and
     then writes results in another stream.)

     Properties are:

    `:coding-system'
          (the symbol of coding system object, which defines the
          decoding.)

    `:stream'
          (the stream on the other end.)

 11. `encoding' (As an input stream, reads data from another stream and
     encodes it according to a coding system.  As an output stream
     encodes the data written to it according to a coding system and
     then writes results in another stream.)

     Properties are:

    `:coding-system'
          (the symbol of coding system object, which defines the
          encoding.)

    `:stream'
          (the stream on the other end.)

   Consider

     (define-stream-type 'type
       :read-function
       :write-function
       :rewind-
       :seek-
       :tell-
       (?:buffer)

   Old Notes:

   Expose lstreams as hash (put get etc. properties) table.

       (write-stream stream string)
       (read-stream stream &optional n sequence)
       (make-stream ...)
       (push-stream-marker stream)
          returns ID prob a stream marker object
       (pop-stream-marker stream)
          backs up stream to last marker
       (unread-stream stream string)
       (stream-available-chars stream)
       (seek-stream stream n)
       (delete-stream stream n)
       (delete-stream-marker stream ic) can always be poe only nested if you
         have set stream marker

       (get-char-stream *generalizes* stream)

       a macro that tries to be efficient perhaps by reading the next
       e.g. 512 characters into a vector and arefing them.  Might check aref
       optimization for vectors in the byte interpreter.

       (make-stream 'process :process ... :type write)

       Consider

       (define-stream-type 'type
         :read-function
         :write-function
         :rewind-
         :seek-
         :tell-
         (?:buffer)


File: internals.info,  Node: Future Work -- Multiple Values,  Next: Future Work -- Macros,  Prev: Future Work -- Lisp Stream API,  Up: Future Work

44.17 Future Work - Multiple Values
===================================

Author: Ben Wing (mailto:ben@xemacs.org)

   On low level, all funs that can return multiple values are defined
with DEFUN_MULTIPLE_VALUES and have an extra parameter, a struct
mv_context *.

   It has to be this way to ensure that only the fun itself, and no
called funs, think they're called in an mv context.

   apply, funcall, eval might propagate their mv context to their
children?

   Might need eval-mv to implement calling a fun in an mv context.
Maybe also funcall_mv? apply_mv?

   Generally, just set up context appropriately.  Call fun (noticing
whether it's an mv-aware fun) and binding values on the way back or
passing them out.  (e.g. to multiple-value-bind)

Common Lisp multiple values, required for specifier improvements.
-----------------------------------------------------------------

The multiple return values from get-specifier should allow the
specifier value to be modified in the correct fashion (i.e.  should
interact correctly with all manner of changes from other callers) using
set-specifier.  We should check this and see if we need other return
values.  (how-to-add? inst-list?)

   In C, call multiple-values-context to get number of expected values,
and multiple-value-set (#, value) to get values other than the first.

   (Returns Qno_value, or something, if there are no values.

   #### Or should throw?  Probably not.  #### What happens if a fn
returns no values but the caller expects a #### value?

   Something like `funcall_with_multiple_values()' for setting up the
context.

   For efficiency, byte code could notice Ffuncall to m.v. functions and
sub in special opcodes during load in processing, if it mattered.


File: internals.info,  Node: Future Work -- Macros,  Next: Future Work -- Specifiers,  Prev: Future Work -- Multiple Values,  Up: Future Work

44.18 Future Work - Macros
==========================

Author: Ben Wing (mailto:ben@xemacs.org)

  1. Option to control whether beep really kills a macro execution.

  2. Recently defined macros are remembered on a stack, so accidentally
     defining another one doesn't fuck you up.  You can "rotate"
     anonymous macros or just pick one (numbered) to put on tags, so it
     works with execute macro - menu shows the anonymous macro, and
     lists some keystrokes.  Normally numbered but you can easily assign
     to named fun or to keyboard sequence or give it a number (or give
     it a letter accelerator?)

