Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 1 of 1
  1. #1
    Senior Coder Rowsdower!'s Avatar
    Join Date
    Oct 2008
    Location
    Some say it's everything.
    Posts
    2,027
    Thanks
    5
    Thanked 397 Times in 390 Posts

    Using cURL to Scan an External Page

    Edit: Nevermind, I figured it out. This is definitely a hosting issue. I started up an account with 000webhost (who advertises availability for cURL) and got the code working. For all interested, it appears as though Freehostia DOES NOT support the use of cURL on their free accounts. Paid accounts may or may not be different...


    I'm messing around trying to scan content from a page on another domain (3rd party, not my site at all) to generate content on my own site on a separate domain based on what I find there (I plan to use a string search on the contents and if certain text is present on the target site I want to change what appears on my site accordingly).

    So I tried print file_get_contents('http://www.domain.com/target/'); and came up completely empty. I'm assuming that there is some kind of issue with the target site using sessions/cookies or similar which is preventing file_get_contents() from working. I read elsewhere online that cURL can be used for this type of thing so I tried this instead as a simple attempt to print the full page contents:

    PHP Code:
    <div style="border:1px solid black;padding:10px;">
    <?php
    function curl_login($url,$data,$proxy,$proxystatus){
        
    $fp fopen("cookie.txt""w");
        
    fclose($fp);
        
    $login curl_init();
        
    curl_setopt($loginCURLOPT_COOKIEJAR"cookie.txt");
        
    curl_setopt($loginCURLOPT_COOKIEFILE"cookie.txt");
        
    curl_setopt($loginCURLOPT_USERAGENT"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
        
    curl_setopt($loginCURLOPT_TIMEOUT40);
        
    curl_setopt($loginCURLOPT_RETURNTRANSFERTRUE);
        if (
    $proxystatus == 'on') {
            
    curl_setopt($loginCURLOPT_SSL_VERIFYHOSTFALSE);
            
    curl_setopt($loginCURLOPT_HTTPPROXYTUNNELTRUE);
            
    curl_setopt($loginCURLOPT_PROXY$proxy);
        }
        
    curl_setopt($loginCURLOPT_URL$url);
        
    curl_setopt($loginCURLOPT_HEADERTRUE);
        
    curl_setopt($loginCURLOPT_USERAGENT$_SERVER['HTTP_USER_AGENT']);
        
    curl_setopt($loginCURLOPT_FOLLOWLOCATIONTRUE);
        
    curl_setopt($loginCURLOPT_POSTTRUE);
        
    curl_setopt($loginCURLOPT_POSTFIELDS$data);
        
    ob_start(); // prevent any output
        
    return curl_exec ($login); // execute the curl command
        
    ob_end_clean(); // stop preventing output
        
    curl_close ($login);
        unset(
    $login);
    }

    function 
    curl_grab_page($site,$proxy,$proxystatus){
        
    $ch curl_init();
        
    curl_setopt($chCURLOPT_RETURNTRANSFERTRUE);
        if (
    $proxystatus == 'on') {
            
    curl_setopt($chCURLOPT_SSL_VERIFYHOSTFALSE);
            
    curl_setopt($chCURLOPT_HTTPPROXYTUNNELTRUE);
            
    curl_setopt($chCURLOPT_PROXY$proxy);
        }
        
    curl_setopt($chCURLOPT_COOKIEFILE"cookie.txt");
        
    curl_setopt($chCURLOPT_URL$site);
        
    ob_start(); // prevent any output
        
    return curl_exec ($ch); // execute the curl command
        
    ob_end_clean(); // stop preventing output
        
    curl_close ($ch);
    }

    curl_login('http://www.domain.com/target/','','','off');
    echo 
    curl_grab_page('http://www.domain.com/target/','','off');

    ?>
    </div>
    The ONLY thing that is returned to the browser from this script for me is the text "29 Naahh.. it will not work!" I don't even see the <div> tags which makes me think that this is some type of host issue (freehostia) that sees the script and refuses to run anything at all. Before I give up, though, I wanted to see if I mangled the code somehow. This is the first time I've heard of or tried cURL so I'm pretty much just copy+pasting and changing the URL.

    Can anyone confirm if freehostia does not allow cURL or if I have the code wrong? If cURL isn't going to work for me, is there any way to use file_get_contents()? Is there something else that I should be trying?
    Last edited by Rowsdower!; 09-10-2009 at 11:40 PM.
    The object of opening the mind, as of opening the mouth, is to shut it again on something solid. –G.K. Chesterton
    See Mediocrity in its Infancy
    It's usually a good idea to start out with this at the VERY TOP of your CSS: * {border:0;margin:0;padding:0;}
    Seek and you shall find... basically:
    validate your markup | view your page cross-browser/cross-platform | free web tutorials | free hosting


 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •