Other problem: caml_apply17

[Alessandro Baretta]

I am also getting this error message: "undefined symbol: caml_apply17". What
library is supposed to define this symbol? Why has it not been exported by the
linker? (Given, of course, that I'm passing -ccopt -Wl,-E to ocamlopt
everywhere--in my code? May I conclude that this symbol is part of a very basic
ocaml library, which MetaOcaml's Makefile does not build or link with -ccopt
-Wl,-E ?

Alex
--
*********************************************************************

Ing. Alessandro Baretta

Studio Baretta
http://studio.baretta.com/

Consulenza Tecnologica e Ingegneria Industriale
Technological Consulting and Industrial Engineering

tel. [phone]
fax. [phone]

[malc]

Post by Alessandro Baretta
I am also getting this error message: "undefined symbol: caml_apply17". What
library is supposed to define this symbol? Why has it not been exported by
the linker? (Given, of course, that I'm passing -ccopt -Wl,-E to ocamlopt
everywhere--in my code? May I conclude that this symbol is part of a very
basic ocaml library, which MetaOcaml's Makefile does not build or link with
-ccopt -Wl,-E ?

caml_applyN etc are generated by ocamlopt when creating startup assembly.

[Alessandro Baretta]

Post by malc

caml_apply17". What library is supposed to define this symbol? Why has
it not been exported by the linker? (Given, of course, that I'm
passing -ccopt -Wl,-E to ocamlopt everywhere--in my code? May I
conclude that this symbol is part of a very basic ocaml library, which
MetaOcaml's Makefile does not build or link with -ccopt -Wl,-E ?

caml_applyN etc are generated by ocamlopt when creating startup assembly.

Ok, so how should I tell gcc to export it int the symtable of the shared object?
And which shared object, by the way?

I have noticed that several caml_applyN symbols are present in the VM
executable, but, obviously not "caml_apply17". So where should I look for this
symbol?

Alex
--
*********************************************************************

Ing. Alessandro Baretta

Studio Baretta
http://studio.baretta.com/

Consulenza Tecnologica e Ingegneria Industriale
Technological Consulting and Industrial Engineering

tel. [phone]
fax. [phone]

[Alessandro Baretta]

Post by Alessandro Baretta

Post by malc

caml_apply17". What library is supposed to define this symbol? Why
has it not been exported by the linker? (Given, of course, that I'm
passing -ccopt -Wl,-E to ocamlopt everywhere--in my code? May I
conclude that this symbol is part of a very basic ocaml library,
which MetaOcaml's Makefile does not build or link with -ccopt -Wl,-E ?

caml_applyN etc are generated by ocamlopt when creating startup assembly.

Ok, so how should I tell gcc to export it int the symtable of the shared
object? And which shared object, by the way?
I have noticed that several caml_applyN symbols are present in the VM
executable, but, obviously not "caml_apply17". So where should I look
for this symbol?

Actually, aside from this issue, the nativized AS/Xcaml is doing a pretty good job.

The random segfaults were due, apparently, to an old shared object which had not
been rebuilt and reinstalled. Now that this problem is solved, most Xcaml code
gets executed without any hassle. Only some Xcaml resources show the caml_applyN
problem.

Alex
--
*********************************************************************

Ing. Alessandro Baretta

Studio Baretta
http://studio.baretta.com/

Consulenza Tecnologica e Ingegneria Industriale
Technological Consulting and Industrial Engineering

tel. [phone]
fax. [phone]

[malc]

Post by Alessandro Baretta

Post by malc

Post by Alessandro Baretta
I am also getting this error message: "undefined symbol: caml_apply17".
What library is supposed to define this symbol? Why has it not been
exported by the linker? (Given, of course, that I'm passing -ccopt -Wl,-E
to ocamlopt everywhere--in my code? May I conclude that this symbol is
part of a very basic ocaml library, which MetaOcaml's Makefile does not
build or link with -ccopt -Wl,-E ?

caml_applyN etc are generated by ocamlopt when creating startup assembly.

Ok, so how should I tell gcc to export it int the symtable of the shared
object? And which shared object, by the way?
I have noticed that several caml_applyN symbols are present in the VM
executable, but, obviously not "caml_apply17". So where should I look for
this symbol?

When you link the final image i.e.:
ocamlopt a.cmx b.cmx c.cmx ...

compiler looks at "non-standard" applyN and curryN functions that are
required and emits startup.s where those apply/curry functions are placed
then it assembles the file and links it together with the rest of the
crowd. There's no specific place you should look for them they can be
anywhere/everywhere/nowhere. nm/objdump/readelf/[ocamlopt -d...] are your
friends.

[Alessandro Baretta]

Post by malc

Post by Alessandro Baretta
Ok, so how should I tell gcc to export it int the symtable of the
shared object? And which shared object, by the way?
I have noticed that several caml_applyN symbols are present in the VM
executable, but, obviously not "caml_apply17". So where should I look
for this symbol?

There's no specific place you should look for them they can be
anywhere/everywhere/nowhere.

This seems to be it! They simply don't exist. They have not been generated. Now
I have to figure out how to force ocamlopt to export *all* of them. What
construct causes ocamlopt to generate such a symbol?

Alex
--
*********************************************************************

Ing. Alessandro Baretta

Studio Baretta
http://studio.baretta.com/

Consulenza Tecnologica e Ingegneria Industriale
Technological Consulting and Industrial Engineering

tel. [phone]
fax. [phone]

[malc]

Post by Alessandro Baretta

Post by malc

Post by Alessandro Baretta
Ok, so how should I tell gcc to export it int the symtable of the shared
object? And which shared object, by the way?
I have noticed that several caml_applyN symbols are present in the VM
executable, but, obviously not "caml_apply17". So where should I look for
this symbol?

There's no specific place you should look for them they can be
anywhere/everywhere/nowhere.

This seems to be it! They simply don't exist. They have not been generated.
Now I have to figure out how to force ocamlopt to export *all* of them. What
construct causes ocamlopt to generate such a symbol?

<moo.ml>
let apply f n1 n2 n3 n4 n5 n6 n7 n8 n9 n10 n11 n12 n13 n14 n15 n16 n17 =
f n1 n2 n3 n4 n5 n6 n7 n8 n9 n10 n11 n12 n13 n14 n15 n16 n17
</moo.ml>

$ ocamlopt -dstartup moo.ml
$ grep caml_apply17 /tmp/camlstartup*.s
.globl caml_apply17
.type caml_apply17,@function
caml_apply17:
$ nm a.out | grep caml_apply17
08049580 T caml_apply17
$ rm -f /tmp/camlstartup*.s

--
mailto:***@pulsesoft.com

[o***@pobox.com]

Hello!

Malc has very well explained the problem and its
solution. Just for the record and future documentation, I'd like to
quote from my message To Walid and Cristiano half a year ago:

There may be a rare condition when dynamic loading fails, if the
generated code uses functions of higher arity or does currying to a
larger extent than that in the main code. This is because
currying/apply_n code is generated in the start-up file. I'd expect
this to be a rare condition that may occur only when the main code is
"too simple" (usually it is never the case because the main code
includes stdlib, and the latter is quite complex already). The problem
will manifest itself is the failure of .! (exception will be
thrown). The workaround is easy: just add to the main code some
functions of higher-arity. You don't have to execute them; they just
have to be present.


I guess some rare problems do actually occur; Murphy law is still in
force...

[Alessandro Baretta]

Post by o***@pobox.com
Hello!
Malc has very well explained the problem and its
solution. Just for the record and future documentation, I'd like to
I guess some rare problems do actually occur; Murphy law is still in
force...

I noticed your note at about the same time I received Malc's message. Actually,
the problem is not rare at all in the AS/Xcaml. The Xcaml compiler might
generate functions with indefinitely long signatures. The current upper bound is
around 20 parameters. I patched Compilenv to generate all caml_apply and
caml_curry function up to an arity of 30. This might become a parameter to the
configure script.

This patch, unluckily, does not and cannot settle the startup.s issue with
dynlinking native code. I have also noticed in your note that dynlinked code has
major restrictions: no new exceptions, no globals. (What are globals by the
way?) This restrictions makes it impossible, in general, to use Natdynlink as a
means to build a dynamic application server, as this requires the ability to
dynlink libraries as needed, which must be entirely general. The dynamism might
be restricted to the actual Xcaml scripts, which have are fairly trivial
signature and usually misbehave. Yet, even in this scenario, I have not been
able to eliminate the segfaults entirely.

Any ideas?

And, BTW, here's my latest patch.

Alex
--
*********************************************************************

Ing. Alessandro Baretta

Studio Baretta
http://studio.baretta.com/

Consulenza Tecnologica e Ingegneria Industriale
Technological Consulting and Industrial Engineering

tel. [phone]
fax. [phone]

[Walid Taha]

Can you find a way to dynamically load these helper functions as well,
rather than have to depend on having a fixed number of them?

Is there a runtime overhead associated with calling dynamically loaded
functions?

Walid.

On Mon, 20 Feb 2006, Alessandro Baretta wrote:

|***@pobox.com wrote:
|> Hello!
|>
|> Malc has very well explained the problem and its
|> solution. Just for the record and future documentation, I'd like to
|> quote from my message To Walid and Cristiano half a year ago:
|>
|> I guess some rare problems do actually occur; Murphy law is still in
|> force...
|
|I noticed your note at about the same time I received Malc's message. Actually,
|the problem is not rare at all in the AS/Xcaml. The Xcaml compiler might
|generate functions with indefinitely long signatures. The current upper bound is
|around 20 parameters. I patched Compilenv to generate all caml_apply and
|caml_curry function up to an arity of 30. This might become a parameter to the
|configure script.
|
|This patch, unluckily, does not and cannot settle the startup.s issue with
|dynlinking native code. I have also noticed in your note that dynlinked code has
|major restrictions: no new exceptions, no globals. (What are globals by the
|way?) This restrictions makes it impossible, in general, to use Natdynlink as a
|means to build a dynamic application server, as this requires the ability to
|dynlink libraries as needed, which must be entirely general. The dynamism might
|be restricted to the actual Xcaml scripts, which have are fairly trivial
|signature and usually misbehave. Yet, even in this scenario, I have not been
|able to eliminate the segfaults entirely.
|
|Any ideas?
|
|And, BTW, here's my latest patch.
|
|Alex
|
|--
|*********************************************************************
|
|Ing. Alessandro Baretta
|
|Studio Baretta
|http://studio.baretta.com/
|
|Consulenza Tecnologica e Ingegneria Industriale
|Technological Consulting and Industrial Engineering
|
|tel. [phone]
|fax. [phone]
|
|
|
|!DSPAM:43f98d28199598298612067!
|

[Alessandro Baretta]

Post by Walid Taha
Can you find a way to dynamically load these helper functions as well,
rather than have to depend on having a fixed number of them?

The point is that they are not even generated by ocamlopt unless ocamlopt sees
the cmxs using them at compile time. A priori, the only caml_applyN functions
ocamlopt always provides are caml_apply2 and caml_apply3. Obviously, most
modules and libraries contain functions whose arity is greater than 3; hence,
unless for every function in the shared object the executable happens to have at
least a function with same arity, we have an unresolved symbol at dynlink time.
This is why I force ocamlopt to generate all caml_applyN and caml_curryN_M
functions up to N=30.

An idea might to be generate the caml_applyN functions once and for all, and put
them all in a shared object, to be loaded before any other shared object

Post by Walid Taha
Is there a runtime overhead associated with calling dynamically loaded
functions?

Well, I suppose non local calls are always a bit more expensive than local ones.
You probably go through a pointer instead of jumping off of a constant symbol.
Also, shared objects are probably less friendly to hierarchical memory than
monolithic executables, on a single process benchmark. In a realistic situation
where different processes share the same "shared" objects, even the memory
bandwidth issue would be to the shared libraries' advantage.

Alex

Post by Walid Taha
Walid.

--
*********************************************************************

Ing. Alessandro Baretta

Studio Baretta
http://studio.baretta.com/

Consulenza Tecnologica e Ingegneria Industriale
Technological Consulting and Industrial Engineering

tel. [phone]
fax. [phone]

[Walid Taha]

Another way of interpreting this situation is that you guys really need to
include some documentation with this file.

Walid.

On Sun, 19 Feb 2006 ***@pobox.com wrote:

|
|Hello!
|
| Malc has very well explained the problem and its
|solution. Just for the record and future documentation, I'd like to
|quote from my message To Walid and Cristiano half a year ago:
|
|There may be a rare condition when dynamic loading fails, if the
|generated code uses functions of higher arity or does currying to a
|larger extent than that in the main code. This is because
|currying/apply_n code is generated in the start-up file. I'd expect
|this to be a rare condition that may occur only when the main code is
|"too simple" (usually it is never the case because the main code
|includes stdlib, and the latter is quite complex already). The problem
|will manifest itself is the failure of .! (exception will be
|thrown). The workaround is easy: just add to the main code some
|functions of higher-arity. You don't have to execute them; they just
|have to be present.
|
|
|I guess some rare problems do actually occur; Murphy law is still in
|force...
|
|_______________________________________________
|MetaOCaml-Hackers-L mailing list
|MetaOCaml-Hackers-***@mailman.rice.edu
|https://mailman.rice.edu/mailman/listinfo/metaocaml-hackers-l
|
|!DSPAM:43f90101122561685929930!
|

[o***@pobox.com]

Hello!

Post by Walid Taha
Can you find a way to dynamically load these helper functions as well,
rather than have to depend on having a fixed number of them?

Well, in principle this is possible. This is quite complex however:
OCaml generates the _assembler_ code of those functions (which is
placed into a start-up file, along with the other stuff).

Post by Walid Taha
Is there a runtime overhead associated with calling dynamically loaded
functions?

No doubt, there is. Currently, btw, we have to invoke the dynamically
loaded code as a `call-back', which includes setting up special
stack frames. There is a lot of indirection involved. It seems there
may be simpler ways (and I have comments to that effect in the code),
whose investigation is left for future work.

Post by Walid Taha
Another way of interpreting this situation is that you guys really need to
include some documentation with this file.

Precisely. Hopefully this discussion will go towards the creation of
this documentation: this discussion is already searchable and
indexable...

Post by Walid Taha
have not been able to eliminate the segfaults entirely.

I should point out the the order of linking libraries into the main
executable matters (as always). In our prior experience with libtk and
gtk, it seems that those tk libraries must come _before_
metanative.cmxa and hookdynlink on the ocamlopt command line. Probably
because these libraries do their own non-trivial initialization and
perhaps dynamic loading as well. I think the current metaocaml script
always places user libraries and files after metanative.cmxa.

Post by Walid Taha
I have also noticed in your note that dynlinked code has major
restrictions: no new exceptions, no globals.

Yes, it is the deliberate restriction to make code simpler (and
because these things can never occur in the MetaOcaml-generated
code). Making this more generic requires some work...

In general, if you find a puzzling segfault, please send me at least
the backtrace in gdb... Incidentally, when you compile the libnatdyn
library (see natdyn/Makefile), you may want to add a flag
-DDYN_DEBUG=1
to CFLAGS, and rebuild the library. The debugging information
(printed on stderr) can be quite helpful in determining the cause (at
least we can see what was loaded to where and so can interpret the
addresses in the backtrace intelligently).

Post by Walid Taha
I patched Compilenv to generate all caml_apply and caml_curry function
up to an arity of 30. This might become a parameter to the configure
script.

That will work, no doubt. However, I'm a bit uneasy to make any
patches to the OCaml system. So far I have avoided patching of the
source code, and I hope that my changes are resilient against future
versions of Ocaml.

This brings the question to Walid: there were some things Ocaml team
and Xavier Leroy can do to make our lives easier. These are quite
small and hopefully uncontroversial things. I have also a few
suggestions stemmed from my work on shift/reset.

[Walid Taha]

Hi Oleg,

First, many thanks for the clarifications on techncial issues. Now for
documentation issues.

On Mon, 20 Feb 2006 ***@pobox.com wrote:

|Walid wrote:
|> Another way of interpreting this situation is that you guys really need to
|> include some documentation with this file.
|
|Precisely. Hopefully this discussion will go towards the creation of
|this documentation: this discussion is already searchable and
|indexable...

The analogy from my daily life seems to be that a notebook that I filled
with notes, little pieces of code, and proofs can be a substitute for a
paper. It's certainly going in the right direction, but what's needed is
something smaller and to the point. (This is only an objection to the
searchable and indexable part).

|This brings the question to Walid: there were some things Ocaml team
|and Xavier Leroy can do to make our lives easier. These are quite
|small and hopefully uncontroversial things. I have also a few
|suggestions stemmed from my work on shift/reset.

Yes, (and we do still want to talk about the latter), but this is a point
related to documentation. All what is needed is a simple and coherent
story that can be communicated to the OCaml group. I know you're
frustrated by this, and it is understandable. Hopefully we'll be able to
address this this Friday.

Walid.