subprocess.Popen hangs in C4D [CLOSED]

On 25/07/2015 at 03:40, xxxxxxxx wrote:

HELP!!!

I have a major problem that I've just spent a week trying to solve to no avail.  I'm suppose to go on sale soon, but this is a show-stopper!

My plugin works fine on Windows, but on a mac the identical code hangs sometimes.
I created an executable with PyInstaller that runs fine.

p = subprocess.Popen(cmd, shell=False, bufsize=0, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds = False)
pickle.dump(data, p.stdin)
output, error_output = p.communicate()

I get my data back with p.communicate() the child process is done, and I can run it over and over again without problem.  That is, until I restart C4D, then 'subprocess.Popen' hangs C4D until I use mac's 'Force Quit.'  Once that happens, trying to launch any executable with Popen, even from the 'Console' inside C4D, hangs C4D.  Restart C4D and still hangs.

Sometimes, I can make it work again by deleting the child executable folder and forcing Popen to throw a file not found error, but then after running the executable again, upon restart, C4D will freeze up if Popen is used again.

I've been trying to get help with this on IRC #python, but everyone who has tried to help is stumped.

It is as if subprocess.Popen is broken.

What is going on?

On 25/07/2015 at 07:00, xxxxxxxx wrote:

Have you closed the file descriptors? Since you tell it not to close them ... that might be a problem.

On 25/07/2015 at 14:38, xxxxxxxx wrote:

I don't see how that could be the problem, aren't they closed automatically when the child process ends and the function that opened it exits?

If I restart the computer, isn't everything reset?

If I type this into the console

import subprocess
subprocess.Popen(["test1"])

C4D locks up and I have to use 'Force Quit' on it.
"test1" should cause it to throw a file not found error.  That's what it does on my Windows machine.

I'm really lost!

On 25/07/2015 at 14:47, xxxxxxxx wrote:

close_fds = False

...was set because on Windows it has to be for the pipe redirect.

Should this be 'True' on the mac for what I'm doing?

If I open a terminal on the mac and run the test above, I get the proper '[Errno2] No such file of directory' error, but from the console in C4D it locks up C4D.
How do I reset things, rebooting isn't working?

Thanks,

Chris

On 26/07/2015 at 03:39, xxxxxxxx wrote:

Niklas,

I can now turn the problem on and off by moving the PyInstaller folder out of the C4D>plugins folder.
It was here: Preferences>MAXON>CINEMA 4...>plugins>MyPlugin>files>libs>PyInstaller
Simply moving it at or above the 'plugins' folder in the path above stops the freezing up.
Why?

Is C4D scanning the plugins folder and finding python files in the PyInstaller folder and screwing things up?

This is so confusing.

Remember the freeze up can be caused by simply calling 'subprocess.Popen(['bad_path_test'])
I don't even have to actually try to use my PyInstaller folder.  Just trying to trigger a file not found error is enough to make it freeze up.  Then, move the PyInstaller folder to 'plugins' or above and you get the file not found error.

What is going on?

On 26/07/2015 at 07:17, xxxxxxxx wrote:

Does it break once you called subprocess.Popen() once? And does it only break when you use it to start
the PyInstaller or if you start any other program? If it only breaks with your PyInstaller thing, could you
send it to me or show me how you created the executable?

But still, this sounds very strange. Sure, the file descriptors should be closed by the OS automatically
when the parent process exits. But they would not if the process changes its parent.. I think.

On 26/07/2015 at 15:29, xxxxxxxx wrote:

Niklas,

Through the painstaking process of elimination, I have sifted through the files created by PyInstaller and discovered that the presence of 4 dynlibs triggers the problem.

1. Place the folder with these 4 files anywhere inside the 'plugins' folder here: Preferences>MAXON>CINEMA 4...>plugins

https://www.dropbox.com/s/ahsowejx862bio1/problem.zip?dl=0

2. Restart C4D;  important - moving the folder in after C4D starts doesn't cause the problem.

3. In the console type: import subprocess.  Then type: subprocess.Popen(['bad_filepath'])
   or any other bad file path.  Or try to launch a program.  It doesn't matter. Just using subprocess.Popen will cause the problem.

Hopefully it will happen on your mac also, remember, I didn't have this problem on Windows.

Let me know,

Chris

On 27/07/2015 at 05:52, xxxxxxxx wrote:

Hi Chris,

I can confirm the lock up on Mac described in your last post.
Unfortunately I'm not sure, there's much I can do about it. Can you perhaps describe the structure of your project? Obviously there are several parts, a plugin a standalone executable, both communicating via a pipe. Do you really need to place the libraries and the executable inside the plugins folder? Just thinking...

On 27/07/2015 at 07:13, xxxxxxxx wrote:

There might be some problems with them being dylibs but not Cinema 4D plugins, yet they reside
in the Cinema 4D plugins folder. Can you change your set-up sligthly, putting the executable and
the libraries into any but the plugins folder and testing again? If the problem persists, there must
another reason for it.

On 27/07/2015 at 12:42, xxxxxxxx wrote:

Andreas and Niklas,

Thank God the problem repeated on your system!
I hate it when people can't duplicate a gremlin.

I have moved my executables to 2 steps above the 'plugins' folder so they are now in the 'Preferences' folder.  My plugin will find them with relative paths starting from the plugin .pypv file, so there shouldn't be a problem.

I would prefer not to do this, and to keep things contained in the main plugin folder, but I guess this isn't possible.

Why is C4D scanning the plugins folder for dynlibs anyway?
There is something wrong that should be addressed in C4D.  This doesn't happen on Windows.
Are you guys going to report this as a bug or should I?

Thanks,

Chris

On 27/07/2015 at 13:27, xxxxxxxx wrote:

Hi Chris,

so I can assume that it doesn't happen now that you moved the dylibs 2 folders above the 'plugins' folder?
Of course this is not very convenient for the user to install, but I don't think there is a workaround for it.

*.dylib is the suffix for Cinema 4D plugins as well as for all other dynamic libraries on Mac OS. On
Windows, they use the distinct *.cdl64 suffix.

On 27/07/2015 at 15:46, xxxxxxxx wrote:

Niklas,

It is working fine, but Maxon needs to fix this problem.
It is so strange, that it took me over a week to understand that I wasn't causing it with my plugin, and to narrow it down.

I probably won't go on sale now in time for the Siggraph convention because of this.

I have automated a solution inside my plugin that moves the files without the user having to do anything.

Are you guys going to report this, or should I?

Thanks for checking on it,

Chris

On 28/07/2015 at 03:33, xxxxxxxx wrote:

Hi Chris,

in general Niklas is right, that Cinema 4D searches the subfolders of "plugins" directory for .dylibs in order to find plugins and I'd like to add, that this is not a general problem. Something seems to be strange with with your specific libraries, as I have numerous other plugins with libraries inside the plugins folder, which don't cause any problems.

Another idea for your project:
There is a number of subdirectory names you can use inside your plugin directory, C4D doesn't touch, when searching for plugins (coffee, res, resource, source, obj, build, api, every subdir beginning with strings_ or _api, every subdir ending on xcodeproj). So maybe this helps in your current situation.

That being said, I will also file a bugreport with your set of libraries. Maybe development can find something. We'll see.