Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 5 of 5
  1. #1
    New Coder
    Join Date
    Aug 2008
    Posts
    12
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Fatal Error when downloading file that is too big. Is there a work around?

    I have a cron job scheduled to download a file from someone else's web server each morning. The job points to a php script with this code
    PHP Code:
    #!/usr/bin/php
    <?php
    $file 
    file_get_contents("http://www.ZZZZ.com/path/info.txt");
    file_put_contents("/myPath/myInfo.txt"$file);
    ?>
    It has worked for smaller files, so I think the error was because this file is so large:
    PHP Fatal error: Allowed memory size of 25165824 bytes exhausted (tried to allocate 18721732 bytes) in /home/myPath/myScript.php on line 4

    Is there a better way to do this?

    Apache version 2.0.63
    PHP version 5.2.5

  • #2
    God Emperor Fou-Lu's Avatar
    Join Date
    Sep 2002
    Location
    Saskatoon, Saskatchewan
    Posts
    16,994
    Thanks
    4
    Thanked 2,662 Times in 2,631 Posts
    Try using standard fread and fwrite techniques for the larger files.
    As long as you're unsetting and overwriting the original variable, you shouldn't have a problem with memory usage.
    Sadly thats about the only idea I can come up with (at least at this time in the morning ) that may work for php, and I can't actually confirm that it will.

    Give it a shot, post back results.

    Edit:
    Oh yes, up you're timelimt to 10 minutes or so max. 30-60 seconds may not be quite enough when you need to use fread and fwrite.
    Last edited by Fou-Lu; 12-19-2008 at 01:12 PM.
    PHP Code:
    header('HTTP/1.1 420 Enhance Your Calm'); 
    Been gone for a few months, and haven't programmed in that long of a time. Meh, I'll wing it ;)

  • Users who have thanked Fou-Lu for this post:

    GameOn (12-19-2008)

  • #3
    New Coder
    Join Date
    Aug 2008
    Posts
    12
    Thanks
    2
    Thanked 0 Times in 0 Posts
    Your idea made reminded me of fputcsv and fgetcsv since I am downloading a csv file. This worked great several times with a file >15MB.

    PHP Code:
    #!/usr/bin/php
    <?php
    $readfile 
    fopen("http://www.ZZZZ.com/path/info.csv","r")  or die("can't open read file");
    $writefile fopen("myPath/myFile.csv","w")  or die("can't open write file");
    $line=fgetcsv($readfile);
    while(! 
    feof($readfile))
    {
      
    fputcsv($writefile,$line,","," ");
      
    $line=fgetcsv($readfile);
    }

    fclose($readfile);
    fclose($writefile);
    ?>
    Oh yes, up you're timelimt to 10 minutes or so max. 30-60 seconds may not be quite enough when you need to use fread and fwrite.
    I am not sure how to do this but it seems like a good safety measure so I will look it up and see if it is something I can add.

  • #4
    New Coder
    Join Date
    Aug 2008
    Posts
    12
    Thanks
    2
    Thanked 0 Times in 0 Posts
    My phpinfo() page is telling me the "max_execution_time = 30" "and safe_mode = off", so I added this line directly below the <?php opening tag...

    PHP Code:
    <?php 
    set_time_limit
    (600);
    The script still works, but of course I cannot confirm the time limit line because it is not taking that long.

    Thanks for the help.

  • #5
    God Emperor Fou-Lu's Avatar
    Join Date
    Sep 2002
    Location
    Saskatoon, Saskatchewan
    Posts
    16,994
    Thanks
    4
    Thanked 2,662 Times in 2,631 Posts
    You're welcome.
    You can test you're timelimit by creating a log file for it. This can actually be done really cheaply (which I won't show you how to do), but its better to use once again an fwrite method:
    PHP Code:
    <?php
    $fStart 
    microtime(true); // PHP 5

    // You're stuff.

    $fEnd microtime(true);
    if (
    $fh fopen('logfile.log''a+'))
    {
        
    $sEntry sprintf("%s took %0.5f seconds\n"date("F j Y H:i:s"), ($fEnd $fStart));
        
    fwrite($fh$sEntry);
        
    fclose($fh);
    }
    Something like this. If you're not on PHP5, microtime will need splitting and combining:
    PHP Code:
    function mtime()
    {
        list(
    $sec$usec) = explode(' ' microtime());
        return ((float)
    $sec + (float)$usec);

    simple as that!
    Of course, you can always just echo out the time it took, but on a cron job you can either send that to you're email or log it (which I prefer).
    PHP Code:
    header('HTTP/1.1 420 Enhance Your Calm'); 
    Been gone for a few months, and haven't programmed in that long of a time. Meh, I'll wing it ;)


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •