[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index][Thread Index][Top&Search][Original]

DynaLoader/MakeMaker problem? - Apache::ASP: crash when placed in startup.pl

From: Alan Burlison <Alan.Burlison@uk.sun.com>
To: Daniel Jacobowitz <drow@false.org>
CC: modperl@apache.org, perl5-porters@perl.org
Date: Sun, 16 Jan 2000 18:26:55 +0000
Message-ID: <38820D6E.86A98632@uk.sun.com>

[p5p folks: I've CC'd this to p5p because I think the proper fix will
require changes to DynaLoader and/or MakeMaker - sorry for the long
post, but the root cause of the problem is somewhat complicated]

Daniel Jacobowitz wrote:

> Ah-HA!
> 
> I believe I see the problem.  In fact, I'm almost sure I do.
> From the one useful ltrace log I was able to get:
> 
[snip]
> 
> Notice that DBI is never dlclose()'d.  But mod_perl is, when apache
> unloads its modules.  The linker is not clever enough to realize that
> DBI depends on symbols in mod_perl.so.  When mod_perl is reloaded at a
> DIFFERENT address, DBI has no reason to be rebound; its symbols are
> already resolved!

That's it! Well done!  I didn't realise that Apache explicitly unloaded
the mod_perl libpel.so.  I've waltzed around this for hours, sure I was
in the right area but never convinced that I had the exact cause
nailed.  That explains all the symptoms, and it also explains why
LD_PRELOAD or adding a linker dependency to the perl XS .so files also
works.  On Solaris when an object is dlclose'd, the linker checks to see
that both it and any modules pulled in by ld.so as the result of a
dependency are not used for resolving any symbols before removing
it/them.  This uses a sort of complicated refcounting mechanism.  Hence
a dlclose() doesn't necessarily mean that the object will be removed.

The mod_perl libperl.so has a dependency on the perl libperl.so.  When
Apache dlopen's the mod_perl libperl.so, ld.so also pulls in the perl
libperl.so.  When Apache dlclose's the mod_perl libperl.so as part of
its restart mechanism, the associated perl libperl.so is also unloaded,
*despite* the fact that it is used to resolve symbols in the perl XS .so
files.

Both LD_PRELOAD and adding a dependency from the perl XS .so files to
the perl libperl.so have the effect of incrementing the refcount of the
perl libperl.so, so when the mod_perl libperl.so is unloaded the perl
libperl.so isn't, as it still has references to it, either as a result
of the LD_PRELOAD, or as a result of the dependency from a perl XS .so
file.

> But now they are resolved to the wrong places.

I've nailed this down exactly.  (I should have done this ages ago) 
Solaris has a nifty feature for auditing the operation of ld.so, so that
you can see the loading and unloading of each .so file.  Note that
tracing calls to dlopen/dlclose is not sufficient on its own - as I said
modules are loaded by ld.so seeing a dependency as well as the direct
result of a dlopen call.

I've instrumented the sequence of .so loads and unloads, and the results
are in two of the attachments on this mail.  'before' is what happens
when Apache SEGVs, and 'after' is what happens with your patch applied.

Basically, in the 'before' case, when Apache dlopen's the mod_perl
libperl.so, ld.so notices the dependency on the perl libperl.so and
pulls it in.  As perl XS .so files are loaded subsequently, they are
linked against the perl libperl.so - all well and good.

Now, Apache restarts, and as a result does a dlclose() on the mod_perl
libperl.so.  ld.so assumes also that the perl libperl.so is no longer
required, so it too is unloaded.

Apache then reopens the mod_perl libperl.so, and as before ld.so notices
the dependency on the perl libperl.so, and loads it back in, BUT AT A
DIFFERENT ADDRESS!  Now the mod_perl libperl.so is correctly linked
agains the perl libperl.so, but all the existing perl XS .so files are
referring to a ghost image of the perl libperl.so that was originally
loaded in.  In the attached "before" file, the perl libperl.so is
originally mapped in at address 0xfee00000, but the second time it is
loaded in at address 0xfec00000 - hence the various symtoms of apparent
memory corruption.

> This looks to be precisely contrary to the way that the linker is
> supposed to work, but I'll get back to that later.

I think it is somewhat of an edge case - you could argue that even
though there is no explicit dependency between the perl XS .so files and
the perl libperl.so, the linker should be able to notice that it has
used symbols from libperl.so and increment the refcounts accordingly. 
Then again, you could also argue that there should be an explicit
dependency between the perl XS .so files and the perl libperl.so.

> Thus my question is, is there a way to tell perl to dlclose() all
> modules?  From looking through the perl code and seeing a noticeable
> lack of dlclose() calls, I'd say no.  DynaLoader doesn't seem to
> support unloading things.  It does, however, store the loaded dlopen
> handles in @DynaLoader::dl_librefs, which we could get at in a cleanup.

I think we have a strong case for:

a) Requesting that MakeMaker adds a dependency between the .so files it
generates and the perl libperl.so

b) Requesting that a 'remove a module' method is added to DynaLoader

I've CCd this to p5p to find out what they think.

> Woah!
> This worked!  Can I get opinions on the attached patch?

It worked for me as well.  One slight concern is that in the attached
"after" file it appears that DynaLoaded .so files are unloaded twice in
succession - I'll try tracking down exactly why this is.

Alan Burlison

--- mod_perl.c.orig	Sat Jan 15 22:08:27 2000
+++ mod_perl.c	Sat Jan 15 22:13:01 2000
@@ -417,23 +417,30 @@
 #if MODULE_MAGIC_NUMBER >= MMN_130
 static void mp_dso_unload(void *data) 
 { 
-    module *modp;
-
-    if(!PERL_DSO_UNLOAD)
-	return;
-
-    if(strEQ(top_module->name, "mod_perl.c"))
-	return;
-
-    for(modp = top_module; modp; modp = modp->next) {
-	if(modp->dynamic_load_handle) {
-	    MP_TRACE_g(fprintf(stderr, 
-			       "mod_perl: cancel dlclose for %s\n", 
-			       modp->name));
-	    modp->dynamic_load_handle = NULL;
+	I32 i;
+	void *handle;
+	AV *librefs;
+	SV *handle_sv;
+	
+	librefs = perl_get_av("DynaLoader::dl_librefs", FALSE);
+	if (!librefs) {
+		MP_TRACE(fprintf(stderr, "Could not get @dl_librefs for unloading.\n"));
+		return;
 	}
-    }
-} 
+	
+	for(i = 0; i <= AvFILL(librefs); i++) {
+		handle_sv = *av_fetch(librefs, i, FALSE);
+		if(!handle_sv) {
+			MP_TRACE(fprintf(stderr, "Could not fetch $dl_librefs[%d]!\n", i));
+			continue;
+		}
+		handle = (void *)SvIV(handle_sv);
+		if(dlclose(handle) < 0) {
+			fprintf(stderr, "Could not close $dl_librefs[%d]: %s\n", i, dlerror());
+			continue;
+		}
+	}
+}
 #endif
 
 static void mp_server_notstarting(void *data)

Linker audit trace without patch applied
========================================

LOAD:   MAP=base  ADDR=0x10000 FILE=./httpd
LOAD:   MAP=ld.so ADDR=0xff3b0000 FILE=/usr/lib/ld.so.1
LOAD:   MAP=base  ADDR=0xff2f0000 FILE=/usr/lib/libsocket.so.1
LOAD:   MAP=base  ADDR=0xff100000 FILE=/usr/lib/libnsl.so.1
LOAD:   MAP=base  ADDR=0xff320000 FILE=/usr/lib/libdl.so.1
LOAD:   MAP=base  ADDR=0xff000000 FILE=/usr/lib/libc.so.1
LOAD:   MAP=base  ADDR=0xff1e0000 FILE=/usr/lib/libmp.so.2
LOAD:   MAP=base  ADDR=0xff1d0000 FILE=/usr/platform/SUNW,Ultra-1/lib/libc_psr.so.1
LOAD:   MAP=base  ADDR=0xfef80000 FILE=/home2/web/apache_1.3.9/libexec/libperl.so
LOAD:   MAP=base  ADDR=0xfee00000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/CORE/libperl.so
LOAD:   MAP=base  ADDR=0xff1b0000 FILE=/usr/lib/libcrypt_i.so.1
LOAD:   MAP=base  ADDR=0xff0e0000 FILE=/usr/lib/libgen.so.1
LOAD:   MAP=base  ADDR=0xfede0000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/Data/Dumper/Dumper.so
LOAD:   MAP=base  ADDR=0xfeda0000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/B/B.so
UNLOAD: MAP=base  ADDR=0xfee00000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/CORE/libperl.so
UNLOAD: MAP=base  ADDR=0xfef80000 FILE=/home2/web/apache_1.3.9/libexec/libperl.so
UNLOAD: MAP=base  ADDR=0xff1b0000 FILE=/usr/lib/libcrypt_i.so.1
UNLOAD: MAP=base  ADDR=0xff0e0000 FILE=/usr/lib/libgen.so.1
UNLOAD: MAP=base  ADDR=0xfef80000 FILE=/home2/web/apache_1.3.9/libexec/libperl.so
UNLOAD: MAP=base  ADDR=0xfee00000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/CORE/libperl.so
UNLOAD: MAP=base  ADDR=0xff1b0000 FILE=/usr/lib/libcrypt_i.so.1
UNLOAD: MAP=base  ADDR=0xff0e0000 FILE=/usr/lib/libgen.so.1
LOAD:   MAP=base  ADDR=0xfef80000 FILE=/home2/web/apache_1.3.9/libexec/libperl.so
LOAD:   MAP=base  ADDR=0xfec00000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/CORE/libperl.so
LOAD:   MAP=base  ADDR=0xff1b0000 FILE=/usr/lib/libcrypt_i.so.1
LOAD:   MAP=base  ADDR=0xff0e0000 FILE=/usr/lib/libgen.so.1
Segmentation Fault(coredump)

Linker audit trace with unload patch applied
============================================

LOAD:   MAP=base  ADDR=0x10000 FILE=./httpd
LOAD:   MAP=ld.so ADDR=0xff3b0000 FILE=/usr/lib/ld.so.1
LOAD:   MAP=base  ADDR=0xff2e0000 FILE=/usr/lib/libsocket.so.1
LOAD:   MAP=base  ADDR=0xff100000 FILE=/usr/lib/libnsl.so.1
LOAD:   MAP=base  ADDR=0xff300000 FILE=/usr/lib/libdl.so.1
LOAD:   MAP=base  ADDR=0xff000000 FILE=/usr/lib/libc.so.1
LOAD:   MAP=base  ADDR=0xff1d0000 FILE=/usr/lib/libmp.so.2
LOAD:   MAP=base  ADDR=0xff1c0000 FILE=/usr/platform/SUNW,Ultra-1/lib/libc_psr.so.1
LOAD:   MAP=base  ADDR=0xfef80000 FILE=/home2/web/apache_1.3.9/libexec/libperl.so
LOAD:   MAP=base  ADDR=0xfee00000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/CORE/libperl.so
LOAD:   MAP=base  ADDR=0xff0e0000 FILE=/usr/lib/libcrypt_i.so.1
LOAD:   MAP=base  ADDR=0xfede0000 FILE=/usr/lib/libgen.so.1
LOAD:   MAP=base  ADDR=0xfedc0000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/Data/Dumper/Dumper.so
LOAD:   MAP=base  ADDR=0xfed80000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/B/B.so
UNLOAD: MAP=base  ADDR=0xfedc0000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/Data/Dumper/Dumper.so
UNLOAD: MAP=base  ADDR=0xfedc0000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/Data/Dumper/Dumper.so
UNLOAD: MAP=base  ADDR=0xfed80000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/B/B.so
UNLOAD: MAP=base  ADDR=0xfed80000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/B/B.so
UNLOAD: MAP=base  ADDR=0xfee00000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/CORE/libperl.so
UNLOAD: MAP=base  ADDR=0xfef80000 FILE=/home2/web/apache_1.3.9/libexec/libperl.so
UNLOAD: MAP=base  ADDR=0xff0e0000 FILE=/usr/lib/libcrypt_i.so.1
UNLOAD: MAP=base  ADDR=0xfede0000 FILE=/usr/lib/libgen.so.1
UNLOAD: MAP=base  ADDR=0xfef80000 FILE=/home2/web/apache_1.3.9/libexec/libperl.so
UNLOAD: MAP=base  ADDR=0xfee00000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/CORE/libperl.so
UNLOAD: MAP=base  ADDR=0xff0e0000 FILE=/usr/lib/libcrypt_i.so.1
UNLOAD: MAP=base  ADDR=0xfede0000 FILE=/usr/lib/libgen.so.1
LOAD:   MAP=base  ADDR=0xfef80000 FILE=/home2/web/apache_1.3.9/libexec/libperl.so
LOAD:   MAP=base  ADDR=0xfee00000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/CORE/libperl.so
LOAD:   MAP=base  ADDR=0xff0e0000 FILE=/usr/lib/libcrypt_i.so.1
LOAD:   MAP=base  ADDR=0xfede0000 FILE=/usr/lib/libgen.so.1
LOAD:   MAP=base  ADDR=0xfedc0000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/Data/Dumper/Dumper.so
LOAD:   MAP=base  ADDR=0xfed80000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/B/B.so
LOAD:   MAP=base  ADDR=0xfed60000 FILE=/usr/lib/nss_files.so.1
UNLOAD: MAP=base  ADDR=0xfedc0000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/Data/Dumper/Dumper.so
UNLOAD: MAP=base  ADDR=0xfed80000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/auto/B/B.so
UNLOAD: MAP=base  ADDR=0xfee00000 FILE=/home1/software/perl/debug/lib/5.00503/sun4-solaris/CORE/libperl.so
UNLOAD: MAP=base  ADDR=0xfef80000 FILE=/home2/web/apache_1.3.9/libexec/libperl.so
UNLOAD: MAP=base  ADDR=0xff2e0000 FILE=/usr/lib/libsocket.so.1
UNLOAD: MAP=base  ADDR=0xfed60000 FILE=/usr/lib/nss_files.so.1
UNLOAD: MAP=base  ADDR=0xff100000 FILE=/usr/lib/libnsl.so.1
UNLOAD: MAP=base  ADDR=0xff1d0000 FILE=/usr/lib/libmp.so.2
UNLOAD: MAP=base  ADDR=0xff0e0000 FILE=/usr/lib/libcrypt_i.so.1
UNLOAD: MAP=base  ADDR=0xfede0000 FILE=/usr/lib/libgen.so.1
UNLOAD: MAP=base  ADDR=0xff000000 FILE=/usr/lib/libc.so.1

/*
 * audit.c
 * ld.so load/unload audit for Solaris
 * Usage:
 *    $ cc -o audit.so -G -K pic -z defs audit.c -lmapmalloc -lc
 *    $ LD_AUDIT=./audit.so <cmd>
 */

#include <link.h>
#include <limits.h>
#include <stdio.h>
  
typedef struct ldlib
   {
   char          name[PATH_MAX];
   char          map[16];
   unsigned long addr;
   }
ldlib_t;

static ldlib_t libs[128];
static int nlibs = 0;

static char *lmid_str(Lmid_t lmid)
{
static char buf[16];
switch (lmid)
   {
   case LM_ID_BASE:
      return("base ");
   case LM_ID_LDSO:
      return("ld.so");
   default:
      sprintf(buf, "#%-4d", lmid);
      return(buf);
   }
}

uint_t la_version(uint_t version)
{
return (LAV_CURRENT);
}
  
uint_t la_objopen(Link_map *lmp, Lmid_t lmid, uintptr_t *cookie)
{
ldlib_t *llp = &(libs[nlibs]);
nlibs++;
strcpy(llp->name, lmp->l_name);
strcpy(llp->map, lmid_str(lmid));
llp->addr = lmp->l_addr;
*cookie = (uintptr_t)llp;
printf("LOAD:   MAP=%s ADDR=0x%p FILE=%s\n", llp->map, llp->addr, llp->name);
fflush(stdout);
return (0);
}

uint_t la_objclose(uintptr_t *cookie)
{
ldlib_t *llp = (ldlib_t*)*cookie;
printf("UNLOAD: MAP=%s ADDR=0x%p FILE=%s\n", llp->map, llp->addr, llp->name);
fflush(stdout);
return(0);
}

Follow-Ups from:: Daniel Jacobowitz <drow@false.org>
Matt Sergeant <matt@sergeant.org>

[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index][Thread Index][Top&Search][Original]