Problem with too many loaded plugins (dlls)

On 08/02/2013 at 02:51, xxxxxxxx wrote:

User Information:
Cinema 4D Version:   All 
Platform:   Windows  ;   
Language(s) :     C++  ;

---------
Hello Maxon, fellow plugin developers,

yesterday Phoenix and I were investigating the problem where, if too many plugins or dlls are loaded, no additional dlls would load.

It came to my attention when some of our dependencies (namely cuda and opencl) stopped working for some users.

We have found that it is related to the fact that virtually all plugins (as well as nvcuda.dll and opencl.dll) are using the statically linked CRT.

It seems a process can only initialise a few static crts before loading additional pluigins or dlls using the static CRT will simply fail (this is a fairly well known problem:
msdn discussion
stackoverflow )

In my tests (I generated 200 simple command plugins with unique plugin IDS) I found that after loading 73 plugins no further plugins can be loaded.

The simple solution to this problem: Using the dynamically linked CRT. When linking all 200 simple command plugins with the dynamic CRT all 200 plugins load (of course the appropriate VC++ redist must be installed on the system or the crt dlls must be in some location C4D is aware of in general)

In VS linking the dynamic CRT can be enabled in the Project settings > "C/C++" > "Code Generation" by setting "Runtime Library" to "Multi-threaded DLL (/MD)".

If you want to test this behaviour yourself, here is the archive with 200 plugins (x64) using the static CRT:
200 plugins static CRT R12+.zip

And here are the same 200 plugins using the dynamic CRT:
200 plugins dynamic CRT R12+.zip
(requires the VC++ 2012 x64 redist)

Unless Maxon advises strictly against using the dynmaic CRT we can only fix this problem if we all link our plugins with the dynamic CRT - it is a massive hassle for all users and I am sure has cause some headache for a few of you already.

On 08/02/2013 at 04:33, xxxxxxxx wrote:

I've asked before in this forum if there was a limit on the number of loaded plugins, and was told there was not, nor on the amount of memory that is available for plugins (other than the system memory limit of course). I ran into this problem on one of mine which loads a number of DLLs.

But there clearly is some kind of limit, and it is seen particularly if the plugins are big ones using lots of memory and/or load a lot of external DLLs. The bigger they are, the more likely you are to run into this problem ISTM.

TBH I've no intention to dynamically link my plugs with the CRT. It's bad enough getting some users up and running as it is (if they can do something wrong on installation, they will!) without having to worry about whether they have got the correct runtimes installed.

One could, of course, argue that 73 plugins are enough for anyone! The ideal IMO would be for Maxon to change the architecture to make it possible to load and unload plugins without having to restart C4D. But I doubt that will happen because it might break all existing plugins!

On 08/02/2013 at 05:56, xxxxxxxx wrote:

Originally posted by xxxxxxxx

<ADDRESS>
User Information:
Cinema 4D Version:   All 
Platform:   Windows  ;   
Language(s) :    
C++  ;

---------
</ADDRESS> Hello Maxon, fellow plugin developers,

yesterday Phoenix and I were investigating the problem where, if too many plugins or dlls are loaded, no additional dlls would load.

It came to my attention when some of our dependencies (namely cuda and opencl) stopped working for some users.

We have found that it is related to the fact that virtually all plugins (as well as nvcuda.dll and opencl.dll) are using the statically linked CRT.

It seems a process can only initialise a few static crts before loading additional pluigins or dlls using the static CRT will simply fail (this is a fairly well known problem:
msdn discussion
stackoverflow )

In my tests (I generated 200 simple command plugins with unique plugin IDS) I found that after loading 73 plugins no further plugins can be loaded.

The simple solution to this problem: Using the dynamically linked CRT. When linking all 200 simple command plugins with the dynamic CRT all 200 plugins load (of course the appropriate VC++ redist must be installed on the system or the crt dlls must be in some location C4D is aware of in general)

In VS linking the dynamic CRT can be enabled in the Project settings > "C/C++" > "Code Generation" by setting "Runtime Library" to "Multi-threaded DLL (/MD)".

If you want to test this behaviour yourself, here is the archive with 200 plugins (x64) using the static CRT:
200 plugins static CRT R12+.zip

And here are the same 200 plugins using the dynamic CRT:
200 plugins dynamic CRT R12+.zip
(requires the VC++ 2012 x64 redist)

Unless Maxon advises strictly against using the dynmaic CRT we can only fix this problem if we all link our plugins with the dynamic CRT - it is a massive hassle for all users and I am sure has cause some headache for a few of you already.

I'll have to check this next week, but I think this could be a limitation of Microsoft's compiler and its lib only.

We're using the same load mechanism internally (as you do when loading dlls), but we're avoiding Microsofts compiler whenever possible and use Intel's compiler and libs instead on Windows (exceptions are codes like ILM's OpenEXR which doesn't compile with Intel's compiler) - and we're already loading more than 64 dlls when Cinema is up and running - without your plugins - which is beyond the lower limit of 64 entries for TLS mentioned in the docs of the linked articles.

If you're willing to deal with the DLL version (and redistributable) trouble, linking dynamically should be ok, but using a different compiler might solve the problem too. I'll keep you updated on that...

Best regards,

Wilfried

On 08/02/2013 at 05:59, xxxxxxxx wrote:

Originally posted by xxxxxxxx

I'll have to check this next week, but I think this could be a limitation of Microsoft's compiler and its lib only.

We're using the same load mechanism internally (as you do when loading dlls), but we're avoiding Microsofts compiler whenever possible and use Intel's compiler and libs instead on Windows (exceptions are codes like ILM's OpenEXR which doesn't compile with Intel's compiler) - and we're already loading more than 64 dlls when Cinema is up and running - without your plugins - which is beyond the lower limit of 64 entries for TLS mentioned in the docs of the linked articles.

If you're willing to deal with the DLL version (and redistributable) trouble, linking dynamically should be ok, but using a different compiler might solve the problem too. I'll keep you updated on that...

Best regards,

Wilfried

Hi Wilfried,

thanks for your reply. I'll wait for your update.

Kind regards,
Yves

On 12/02/2013 at 08:53, xxxxxxxx wrote:

Originally posted by xxxxxxxx

Originally posted by xxxxxxxx

I'll have to check this next week, but I think this could be a limitation of Microsoft's compiler and its lib only.

We're using the same load mechanism internally (as you do when loading dlls), but we're avoiding Microsofts compiler whenever possible and use Intel's compiler and libs instead on Windows (exceptions are codes like ILM's OpenEXR which doesn't compile with Intel's compiler) - and we're already loading more than 64 dlls when Cinema is up and running - without your plugins - which is beyond the lower limit of 64 entries for TLS mentioned in the docs of the linked articles.

If you're willing to deal with the DLL version (and redistributable) trouble, linking dynamically should be ok, but using a different compiler might solve the problem too. I'll keep you updated on that...

Best regards,

Wilfried

Hi Wilfried,

thanks for your reply. I'll wait for your update.

Kind regards,
Yves

After debugging it, I'm not sure that linking the static lib is the root of the problem. There are currently up to 1088 TLS entries per process and what I could see by debugging the crt code, only one of them is used per dll (with a static linked lib).

Furthermore _after_ loading of your plugins failed (about 70 had been loaded), Cinema loaded 15 other plugins that I had installed in my plugins folder, which makes me think, that there 's something going wrong in your dll init code or that we've another (yet unknown) resource allocation or leaking problem that makes the LoadLibrary call fail when loading your plugin.

If you're interested, PM me and send me the plugin code.

Best regards,

Wilfried

On 12/02/2013 at 09:10, xxxxxxxx wrote:

Originally posted by xxxxxxxx

After debugging it, I'm not sure that linking the static lib is the root of the problem. There are currently up to 1088 TLS entries per process and what I could see by debugging the crt code, only one of them is used per dll (with a static linked lib).

Furthermore _after_ loading of your plugins failed (about 70 had been loaded), Cinema loaded 15 other plugins that I had installed in my plugins folder, which makes me think, that there 's something going wrong in your dll init code or that we've another (yet unknown) resource allocation or leaking problem that makes the LoadLibrary call fail when loading your plugin.

If you're interested, PM me and send me the plugin code.

Best regards,

Wilfried

Hi,

thanks for your reply.

This problem happens to other people using actual real world plugins (after all, that's why I started investigating this issue). Also, I have not a single plugin in my collection that still loads _after_ failing to load all 200 test plugins (are you sure they didn't load before?).

I used R12 to test this.

All plugins are variations of this one (with increasing plugin IDs) :

  
  
#include "c4d.h"   
  
#include "c4d_symbols.h"   
  
  
class ExampleCommand : public CommandData   
{   
     public:   
          virtual Bool Execute(BaseDocument *doc);   
};   
  
  
Bool ExampleCommand::Execute(BaseDocument *doc)   
{   
     GePrint("Just a simple example.");   
  
     return TRUE;   
};   
  
  
// be sure to use a unique ID obtained from www.plugincafe.com   
#define ID_EXAMPLECOMMAND 1040001   
  
  
Bool RegisterExampleCommand(void)   
{   
     String name = "Plugin #0001";   
#if CMAKE_C4D_VERSION >= 1200   
     return RegisterCommandPlugin(ID_EXAMPLECOMMAND, name, 0, AutoBitmap("About.tif"), "ExampleCommand", gNew ExampleCommand);   
#else   
     return RegisterCommandPlugin(ID_EXAMPLECOMMAND, name, 0, "About.tif", "ExampleCommand", gNew ExampleCommand);   
#endif   
}   
  
  
  
Bool PluginStart(void)   
{   
  
     // command   
     if (!RegisterExampleCommand()) return FALSE;   
        
     return TRUE;   
}   
  
  
void PluginEnd(void)   
{   
  
}   
  
  
Bool PluginMessage(LONG id, void *data)   
{   
     switch (id)   
     {   
          case C4DPL_INIT_SYS:   
               if (!resource.Init()) return FALSE; // don't start plugin without resource   
               return TRUE;   
  
          case C4DMSG_PRIORITY:   
               return TRUE;   
     }   
  
     return FALSE;   
}   
  

I'll PM you the full thing in a few minutes.

On 12/02/2013 at 09:19, xxxxxxxx wrote:

Originally posted by xxxxxxxx

Originally posted by xxxxxxxx

After debugging it, I'm not sure that linking the static lib is the root of the problem. There are currently up to 1088 TLS entries per process and what I could see by debugging the crt code, only one of them is used per dll (with a static linked lib).

Furthermore _after_ loading of your plugins failed (about 70 had been loaded), Cinema loaded 15 other plugins that I had installed in my plugins folder, which makes me think, that there 's something going wrong in your dll init code or that we've another (yet unknown) resource allocation or leaking problem that makes the LoadLibrary call fail when loading your plugin.

If you're interested, PM me and send me the plugin code.

Best regards,

Wilfried

Hi,

thanks for your reply.

This problem happens to other people using actual real world plugins (after all, that's why I started investigating this issue). Also, I have not a single plugin in my collection that still loads _after_ failing to load all 200 test plugins (are you sure they didn't load before?).

I'll PM you the full thing in a few minutes.

Yes I am - I single step-ed through it and had a look at each of the successful loads (after your plugins failed) and their init code (had been our code and therefore it was easy to do :-)). Please don't forget the project file you're using.

I don't doubt there is a problem (otherwise I hadn't asked you), but I think we haven't found the root of the problem yet.

Best regards,

Wilfried

On 12/02/2013 at 09:22, xxxxxxxx wrote:

That is quite strange. I can give you an archive of plugins that won't load on my machine after the 200 test plugins as well, if you want.

On 12/02/2013 at 09:32, xxxxxxxx wrote:

Originally posted by xxxxxxxx

That is quite strange. I can give you an archive of plugins that won't load on my machine after the 200 test plugins as well, if you want.

Please upload just on of the VS proj files (e.g. the one for plugin 101) that fails to load (solution isn't necessary - I can add that proj to mine manually).

Best regards,

Wilfried

On 12/02/2013 at 11:03, xxxxxxxx wrote:

There you go:

http://www.mediafire.com/?8bjd3qy75r7hr41

Regards,
Yves

On 24/02/2013 at 07:10, xxxxxxxx wrote:

i wrote about something http://forums.cgsociety.org/showthread.php?p=6807775#post6807775
But it happens today. I switch win8 x64(8gbRAM)

On 02/12/2013 at 15:13, xxxxxxxx wrote:

Any news about this topic?