How to download a file from the internet? [SOLVED]

On 22/09/2014 at 13:51, xxxxxxxx wrote:

User Information:
Cinema 4D Version:   14 
Platform:   Windows  ;   Mac OSX  ; 
Language(s) :     C++  ;

---------
I performed a search and found here a similar question but the answers didn't helped me much.
So, here is goes:

How can I download a file from a URL?
It will be a simple .txt file.

And I need to make it work on Mac OS X, Win32 and Win64 (I read on the other post that it could be hard to make it work on Win64).

I also searched the SDK but, since I don't know what to look for, I just found GeOpenHTML and that doesn't help me.

On 22/09/2014 at 14:48, xxxxxxxx wrote:

Let me add that I want to download a file from a username/password protected URL address that I have in my own site.

On 23/09/2014 at 04:30, xxxxxxxx wrote:

I already managed to do it in python.
Damn!! It is soooo easy in python.
How come it is something so complicated in C++?
Could, at least, someone point me to the correct libraries to use?

On 23/09/2014 at 15:17, xxxxxxxx wrote:

Anyone? Please?

On 23/09/2014 at 16:20, xxxxxxxx wrote:

I can't stand to see a grown man beg. :wink:
Here's how I've done it with the Windows standard libray.

//This code downloads a file from the internet using the windows OS  
//It also shows how to convert from a string to a LPCSTR file type  
  
  
#include <string>  
#include <urlmon.h>                 //Needed for the URLDownloadToFile() function  
#pragma comment(lib, "urlmon.lib")  //Needed for the URLDownloadToFile() function  
using namespace std;  
  
int main()  
{  
  string url = "http://www.myurl/mytext.txt";  
  
  //Convert the string to a LPCSTR type so we can use it in the URLDownloadToFile() function  
  LPCSTR lpcURL = url.c_str();  
  
  string destination = "C:\\Users\\user\\Desktop\\webfile.txt";  
  
  //Convert the string to a LPCSTR type so we can use it in the URLDownloadToFile() function  
  LPCSTR lpcDestination = destination.c_str();  
  
  HRESULT hr = URLDownloadToFile( NULL, lpcURL, lpcDestination, 0, NULL );  
  
  return 0;  
}

Hope that helps.

-ScottA

On 24/09/2014 at 01:14, xxxxxxxx wrote:

Thank you, Scott.

I have to cry more often ;-)

I will give it a try. But, just looking at it, will this allow me to download from a web folder that is username/password protected?
Also, does it always download to a file?
In python it opens the internet location as a stream and I can easily download it to a string and then, do whatever I want with it.

On 24/09/2014 at 07:40, xxxxxxxx wrote:

No idea.
I needed a way to download a text file from the web using C++ a couple of years ago. So I went searching for the code to do that. And this is what I found.
I tried it and it worked. Then I never used it again.

I do recall that this code was hard to find.
Most of the really robust URL stuff for C++ seems to be sold in commercial packages.

-ScottA

On 24/09/2014 at 11:08, xxxxxxxx wrote:

Oh, man!!!
Python makes this so, so easy.
Well, I will search more and more.
Thank you, Scott.

On 24/09/2014 at 22:25, xxxxxxxx wrote:

You might want to take a look at libcurl.

http://curl.haxx.se/libcurl/

And if you like the C++ way of doing things then also curlpp, which makes use of libcurl.

https://code.google.com/p/curlpp/

I wrote a blog post about this years ago when I was writing a system to download files from cloud services. This may help you on your way to compiling both libcurl and curlpp.

http://www.gamelogicdesign.com/2012/05/21/a-c-library-for-communicating-with-amazon-s3/

Once you have libcURL integrated into your plugin then you issue HTTP requests (PUT, POST, GET etc..) to communicate with the webservice (website) you are connecting with. So for your situation you would send a HTTP POST with the login and password, get a message back to tell you if that was successful or not and then you can make another call to GET a file.

Kent

On 25/09/2014 at 01:47, xxxxxxxx wrote:

Thank you for the answer, Kent.
Is this cross platform?
I mean, I usually develop in Xcode and then send my source code to Visual Studio to compile for Win32 and Win64.

On 25/09/2014 at 04:09, xxxxxxxx wrote:

I already found out that I have "curl" included in Xcode.
So, I should use that.
However, all the samples (not that many, actually) that I find online use the curl library to download a URL directly to a file.
I don't want that. I want to do it like I do in python. I simply need to read the content of a small, simple .txt file that is online. Just 10 characters.
So, what I want is to access a .txt file that is inside a username/password protected folder.
I have the URL address, the username and the password. I just need to read the content of the file in the URL address.
Why is it that all the C++ code I find is soooooooo complicated?
My python code is:

  
import urllib2   
from urllib2 import URLError, HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler, install_opener, build_opener   
  
def check_online(url,username,password) :   
        
     try:   
          password_mgr.add_password(None, url, username, password)   
          opener = build_opener(HTTPBasicAuthHandler(password_mgr))   
          file = opener.open(url+"/myfile.txt")   
          content=file.read()   
          file.close()   
  
     except:   
          content=""   
  
     return content   

So, so simple. Why must it be so complicated in C++?

On 25/09/2014 at 09:27, xxxxxxxx wrote:

Well, I reached as far as:

  
#include <stdio.h>   
#include <curl/curl.h>   
  
CURL *curl;   
CURLcode res;   
char* usrpass="my_username:my_password";   
char* pointer="1234567890";   
        
curl = curl_easy_init();   
if(curl) {   
     curl_easy_setopt(curl, CURLOPT_URL, "http://my_url_pointing_to_the_file");   
     curl_easy_setopt(curl, CURLOPT_USERPWD , usrpass);   
  
     curl_easy_setopt(curl, CURLOPT_READDATA, pointer);   
     curl_easy_setopt(curl, CURLOPT_READFUNCTION, NULL);   
             
     /* Perform the request, res will get the return code */   
     res = curl_easy_perform(curl);   
     /* Check for errors */   
     if(res != CURLE_OK)   
          fprintf(stderr, "curl_easy_perform() failed: %s\n",   
                         curl_easy_strerror(res));   
             
     /* always cleanup */   
     curl_easy_cleanup(curl);   
}   

But, when building, I get:

Undefined symbols for architecture x86_64:
"_curl_easy_init", referenced from:
      PluginStart() in main.o
"_curl_easy_setopt", referenced from:
      PluginStart() in main.o
"_curl_easy_perform", referenced from:
      PluginStart() in main.o
"_curl_easy_strerror", referenced from:
      PluginStart() in main.o
"_curl_easy_cleanup", referenced from:
      PluginStart() in main.o
ld: symbol(s) not found for architecture x86_64

Also, I get "Conversion from string literal to 'char *' is deprecated"

Can someone help me out with this?

On 25/09/2014 at 18:39, xxxxxxxx wrote:

The reason that C++ is 'so complicated' is that it is literally the OOP extension of C which is a very low-level procedural language.  C is basically a step above Assembler (directly programming the CPU) and C++ is not much further away.  Remember that Python is interpreted while C++ is compiled into executable code (directly executable on the CPU).  The choice of C/C++ is speed since executable code is almost always faster than interpreted code.

On the issue with curl, have you included a lib or dylib to the project?

On 25/09/2014 at 22:10, xxxxxxxx wrote:

I installed curl and your code works fine for me.
All I had to do was change the URL to some text file on Google that I could grab as a test.
I'm guessing that you probably unzipped the curl files to your HD. But you never actually built them?
curl needs to be "built" so that you get a .dll from it.
The instructions for building are in: winbuild/BUILD.WINDOWS.txt
I built mine using this: nmake /f Makefile.vc mode=dll

When you build curl it creates a new folder called "builds"
In there you'll find a bin folder with libcurl.dll in it.
That .dll file has to be copied to your VS project's Debug and Release folders or else they won't run.
QT users know about this sort of thing because they also need to copy some QT .dlls to their plugin's output folders if not using static builds.
It's a fairly common practice. So try to get comfortable with it.

Also. You have to set up your project's properties in VS:
-In the VC++ Directories section you have to set the paths to your curl folders in the "Include Directories" and "Library Directories".
-In the Linker->Input section. You have to open the "Additional Dependencies" option.
And paste this into the top window: libcurl.lib
BTW: If you're wondering where I got libcurl.lib from. I got it from the builds/lib folder which you should see after you build it.

This is similar to how I make all of my C4D plugins too.
I do it all from scratch. I don't copy and paste and then delete what's not needed as Maxon recommends.
I do it all completely from scratch every single time. Because when you download libraries from the web. You'll need to know how all the setting work in VS.
There's no such thing as copy and paste out there in the "real world". :wink:

One thing that you're going to notice is that people who write C++ code rarely communicate clearly to the average user.
And people who write these libraries are the worst at communication.
They have coded stuff for so long that they have completely forgotten what it's like to be a new or casual C++ user.

This is why C++ is thought of as "too hard".
It's not hard at all. It's only hard when people can't communicate clearly and properly.
There's files all over the place, and several things needing to be set up in VS for it all to work.
This stuff has to be communicated to the users clearly. And C++ programmers are the worst of the worst when it comes to communicating things clearly.

-ScottA

On 25/09/2014 at 23:55, xxxxxxxx wrote:

I have noticed that information is very unclear.
Cryptic sentences and, above all, almost all assume that is a veteran programmer reading them. Newbies try to read them and can't make anything out of it.
I have dozens of browser tabs with stuff related to curl but most of them I can't decipher properly :-(

Well, I guess I have something to start with, now. Oh, by the way, my code was mostly what I could come up with after reading a LOT about curl.

However, I start my development in Xcode. I assume that I need to compile the curl library for Mac and compile for Windows too, when I take my code to VS.
So, I still need to find out how to compile it for Mac, first.
Actually, on my Mac, I was just including the curl library that is already in my system. But I guess it is not enough.
Well, off I go, trying to find out how to compile it for Mac.

On 03/10/2014 at 03:21, xxxxxxxx wrote:

Hello again, Scott.

I have it all working fine now in MacOS X. It was not easy but it is working perfectly now.
Now I was trying to build the Win32 and Win64 versions.
So, while trying to build the curl libraries, I got this:

c:\Users\ruibatista\Desktop\curl-7.38.0\winbuild>nmake Makefile.vc mode=dll VC=10

Microsoft (R) Program Maintenance Utility Version 10.00.30319.01
Copyright (C) Microsoft Corporation. All rights reserved.

configuration name: libcurl-vc10-x86-release-dll-ipv6-sspi-winssl
        cl.exe /O2 /DNDEBUG /MD /I. /I ../lib /I../include /nologo /W3 /EHsc /DW
IN32 /FD /c /DBUILDING_LIBCURL /I"../../deps/include" /DUSE_WIN32_IDN /DWANT_ID
N_PROTOTYPES /DUSE_IPV6 /DUSE_WINDOWS_SSPI /DUSE_SCHANNEL /Fo"..\builds\libcur
l-vc10-x86-release-dll-ipv6-sspi-winssl-obj-lib/file.obj" ..\lib\file.c
file.c
../include\curl/curlbuild.h(152) : fatal error C1083: Cannot open include file:
'sys/socket.h': No such file or directory
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0
\VC\BIN\cl.exe"' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0
\VC\BIN\nmake.exe"' : return code '0x2'
Stop.

Must I have the curl folder in some specific place?
Compiling curl in MacOS was so easy. I only had to type one command and everything worked fine the first time :-(

On 03/10/2014 at 03:23, xxxxxxxx wrote:

Also, one of the options is:

MACHINE=<x86 or x64>

Does this mean that, if I want to build Win32 and Win64 version I have to perform two compilations? Will they not conflict?

(but this only matters if I manage to compile it, of course, and I can't even do that now)

On 03/10/2014 at 08:01, xxxxxxxx wrote:

The 32 bit version of anything will typically compile fine when your compiler is making a 64 bit program. So it should be safe to set that option to x64.  Assuming that you're building a 64 bit plugin.
If you're configuration is set to build a Debug | 64 or Release | 64 plugin. Then that option needs to be set to x64.
If you're configuration is set to build a Debug | Win32 or Release | Win32 plugin. Then that option needs to be set to Win32.

The one I installed was: curl 7.38.0. It wasn't the MSVC version.  
It's ok to rename it to "curl" to make the path shorter and easier to use.

Here's the instructions I wrote for myself in case I forget how I did it

Installation instructions  
///////////////////////////////////////////////////////////////////////////////////////////////////////////  
  
-Unzip to c:\curl  
-Build curl to generate the .dll and .lib files needed  
 I used this in the VC console to build it: nmake /f Makefile.vc mode=dll  
  
  
-Set up VS to use them:  
  Put libcurl.lib in the Linker->Input "Additional Dependencies" top window  
  Put C:\curl\builds\libcurl-vc-x86-release-dll-ipv6-sspi-winssl\include\curl   in "Include Directories"   
  Put C:\curl\builds\libcurl-vc-x86-release-dll-ipv6-sspi-winssl\lib            in "Library Directories"  
  
  
  
//////////////////////////////////////////////////////////////////////////////////////////////////////////////  
  
Using the examples  
  
  
-If you can't run your code because of a "libcurl.dll not found" error  
 You might need to copy the builds/bin/libcurl.dll file to your Degug & Release folders to fix that  
  
  
  
NOTE: In VS you will get lots of 'cannot convert from void..' errors from the examples  
    This is because they need to be casted to work  
    Examples: chunk.memory = (char* )malloc(1);  
              mem->memory = (char* )realloc(mem->memory, mem->size + realsize + 1);

-ScottA

On 03/10/2014 at 11:13, xxxxxxxx wrote:

Once again, thank you. I managed to do it with a installer from here (http://www.confusedbycode.com/curl/)
But, if it weren't for you indications, I would never guess that those changes in the Prject Properties were required.
However, now I'm getting these errors:

1>c:\program files (x86)\microsoft visual studio 10.0\vc\include\xlocale(323) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
1>c:\program files (x86)\microsoft visual studio 10.0\vc\include\xlocale(323) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2143: syntax error : missing ')' before 'constant'
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2143: syntax error : missing ';' before 'constant'
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2059: syntax error : ')'
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2238: unexpected token(s) preceding ';'
1>
1>Build FAILED.
1>
1>Time Elapsed 00:00:02.10
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

I didn't included and <msxml.h>
How come it is returning those errors?

On 03/10/2014 at 11:54, xxxxxxxx wrote:

I'm afraid I have no idea about those.

One thing that I've noticed is that VS will throw missing ';' or missing ')';  error messages if you forget the closing brace '}' somewhere in your code.
VS apparently doesn't know how to check your scoping {} braces very well. And it throws those kinds of misleading error messages. And sends you on a wild goose chase if you forgot to close your scope with an '}' in your code.

I doubt that's why you're getting those error though.

-ScottA