Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 10 of 10
  1. #1
    Senior Coder timgolding's Avatar
    Join Date
    Aug 2006
    Location
    Southampton
    Posts
    1,519
    Thanks
    114
    Thanked 110 Times in 109 Posts

    Angry Script not completing

    Hi,

    I have a php cron script that is converting a CSV to a mysql database table. Though i am echoing some data from the script when i run it nothing is being displayed. The script runs and completed most of the inserts but it doesnt finish all the way. I get a empty page returned when it is done. I have set time limit to take as long as it needs with

    PHP Code:
    set_time_limit(0); 
    The script doesn't display any errors it just returns a empty. What could cause the script to halt half way and display no errors even when no time limit is set?
    You can not say you know how to do something, until you can teach it to someone else.

  • #2
    Senior Coder angst's Avatar
    Join Date
    Apr 2004
    Location
    Toronto, Ontario
    Posts
    2,114
    Thanks
    15
    Thanked 122 Times in 122 Posts
    could you post your script?

  • #3
    Senior Coder timgolding's Avatar
    Join Date
    Aug 2006
    Location
    Southampton
    Posts
    1,519
    Thanks
    114
    Thanked 110 Times in 109 Posts
    PHP Code:
    <?php
    // convert all data feeds to MySQL databases for improved efficiency.
    set_time_limit(0);

    if(
    $_SERVER['REMOTE_ADDR']!="192.168.84.21")
        exit;
        
    require(
    "../../config/config2.php");
    require_once(
    "/home/shared_php_lib_v2/database_objects/mysql_connect_class.php");
    require_once(
    "/home/shared_php_lib_v2/database_objects/query_class.php");
    require_once(
    "/home/shared_php_lib_v2/form_objects/form_class.php");
    require_once(
    "/home/shared_php_lib_v2/admin_objects/admin_class.php");

    $admin = new admin($webpage);

    // DEALS SALT
    $query "DROP TABLE IF EXISTS `DEALS_SALT`";
    if(!
    $result $admin->do_task($query"convert_2db.php: Drop table DEALS_SALT"))
    {
        echo 
    "drop table error<br />";
    }
    else
         echo 
    "DEALS_SALT table dropped<br />";

    $query "CREATE TABLE `DEALS_SALT` (
      `KEY` INT( 10 ) UNSIGNED ZEROFILL NOT NULL AUTO_INCREMENT PRIMARY KEY ,
      `SKU` varchar(255) NOT NULL default '',
      `HSNUM` int(10) NOT NULL default '0',
      `TARIFFNUM` int(10) NOT NULL default '0',
      `PRICE` float NOT NULL default '0',
      `FLR` tinyint(2) NOT NULL default '0',
      `HPLR` tinyint(2) NOT NULL default '0',
      `CASH_BACK` float NOT NULL default '0',
      `GIFT` varchar(255) NOT NULL default '',
      `RET` varchar(255) NOT NULL default '',
      `CASH_TYPE` tinyint(1) NOT NULL default '0',
      `CLEARANCE` tinyint(1) NOT NULL default '0',
      `STOCK` tinyint(1) NOT NULL default '0',
      `TOTAL_COST` float NOT NULL default '0',
      `EMLR` float NOT NULL default '0',
      `SKUPOPMONTH` int(10) NOT NULL default '0',
      `DEALSRES1` varchar(255) NOT NULL default '',
      `DEALSRES2` varchar(255) NOT NULL default '',
       KEY `HSNUM` (`HSNUM`)
    ) ENGINE=MyISAM;
    "
    ;
    if(!
    $result $admin->do_task($query"convert_2db.php: Create table DEALS_SALT"))
    {
       echo 
    "create table error<br />";
    }
    else
        echo 
    "DEALS_SALT table created<br />";
        
    $flag true;
    $i=1;
    $fname="part.".$i.".txt";

    while(
    file_exists("../../data/dealsalt/".$fname))
    {
        
    $data get_feed("dealsalt/".$fname);
        
    //
        
    foreach($data as $dat)
        {
                    
    //INSERT INTO tbl_name (a,b,c) VALUES(1,2,3,4,5,6,7,8,9);
                    
    $sql "INSERT INTO `DEALS_SALT` (`SKU`,`HSNUM`,`TARIFFNUM`,`PRICE`,`FLR`,`HPLR`,`CASH_BACK`,`GIFT`,`RET`,`CASH_TYPE`,`CLEARANCE`, `STOCK`, `TOTAL_COST`, `EMLR`, `SKUPOPMONTH`, `DEALSRES1`, `DEALSRES2`)
                    VALUES('"
    .mysql_real_escape_string($dat[0])."', '".mysql_real_escape_string($dat[1])."','".mysql_real_escape_string($dat[2])."','".mysql_real_escape_string($dat[3])."','".mysql_real_escape_string($dat[4])."','".mysql_real_escape_string($dat[5])."', '".mysql_real_escape_string($dat[6])."','".mysql_real_escape_string($dat[7])."','".mysql_real_escape_string($dat[8])."','".mysql_real_escape_string($dat[9])."','".mysql_real_escape_string($dat[10])."', '".mysql_real_escape_string($dat[11])."','".mysql_real_escape_string($dat[12])."','".mysql_real_escape_string($dat[13])."','".mysql_real_escape_string($dat[14])."','".mysql_real_escape_string($dat[15])."','".mysql_real_escape_string($dat[16])."')"
                    if(!
    $result $admin->do_task($sql"convert_2db.php: Insert into table DEALS_SALT"))
                    {
                        echo 
    "error "$sql "<br />";
                        
    $flag false;
                    }
            
        }
        
    $i++;
        
    $fname="part.".$i.".txt";
    }
    if(
    $flag)
        echo 
    "DEALS_SALT data inserted<br />";


    $admin->db->db_close();

    function 
    get_feed($file)
    {
        
    $data = array();
        
    $data_lines file("../../data/".$file);
        foreach(
    $data_lines as $dat)
        {
            
    array_push($dataexplode("\t"$dat));
        }
        return 
    $data;   
    }
    ?>
    You can not say you know how to do something, until you can teach it to someone else.

  • #4
    Regular Coder
    Join Date
    Dec 2009
    Location
    UK
    Posts
    495
    Thanks
    0
    Thanked 58 Times in 58 Posts
    add error_reporting(E_ALL); to the top of the script in case it's got error messages turned off
    My site: JayGilford.com
    Resources:
    PHP Pagination Class | Getting all page links | Handling PHP Errors properly
    If you like a users help, show your appreciation with the rep and thanks buttons :)

  • #5
    Regular Coder ninnypants's Avatar
    Join Date
    Apr 2008
    Location
    Utah
    Posts
    504
    Thanks
    10
    Thanked 47 Times in 47 Posts
    Your problem is this line
    PHP Code:
    set_time_limit(0;) 
    http://us.php.net/manual/en/function.set-time-limit.php

    you are giving your script 0 seconds to run so it is probably thowing a fatal error as soon as it reaches 1 second of execution time

  • #6
    God Emperor Fou-Lu's Avatar
    Join Date
    Sep 2002
    Location
    Saskatoon, Saskatchewan
    Posts
    16,994
    Thanks
    4
    Thanked 2,662 Times in 2,631 Posts
    Quote Originally Posted by ninnypants View Post
    Your problem is this line
    PHP Code:
    set_time_limit(0;) 
    http://us.php.net/manual/en/function.set-time-limit.php

    you are giving your script 0 seconds to run so it is probably thowing a fatal error as soon as it reaches 1 second of execution time
    No, 0 indicates no time limit imposed.
    Try putting more weight against you're sql instead of you're php code. Develop you're insertions to take multiple inserts on a single execution:
    PHP Code:
        $insert = array();
        foreach(
    $data as $dat
        { 
                    
    //INSERT INTO tbl_name (a,b,c) VALUES(1,2,3,4,5,6,7,8,9); 
                    
    $insert[] = "('".mysql_real_escape_string($dat[0])."', '".mysql_real_escape_string($dat[1])."','".mysql_real_escape_string($dat[2])."','".mysql_real_escape_string($dat[3])."','".mysql_real_escape_string($dat[4])."','".mysql_real_escape_string($dat[5])."', '".mysql_real_escape_string($dat[6])."','".mysql_real_escape_string($dat[7])."','".mysql_real_escape_string($dat[8])."','".mysql_real_escape_string($dat[9])."','".mysql_real_escape_string($dat[10])."', '".mysql_real_escape_string($dat[11])."','".mysql_real_escape_string($dat[12])."','".mysql_real_escape_string($dat[13])."','".mysql_real_escape_string($dat[14])."','".mysql_real_escape_string($dat[15])."','".mysql_real_escape_string($dat[16])."')";
        }
        
    $sql "INSERT INTO `DEALS_SALT` (`SKU`,`HSNUM`,`TARIFFNUM`,`PRICE`,`FLR`,`HPLR`,`CASH_BACK`,`GIFT`,`RET`,`CASH_TYPE`,`CLEARANCE`, `STOCK`, `TOTAL_COST`, `EMLR`, `SKUPOPMONTH`, `DEALSRES1`, `DEALSRES2`) 
                    VALUES " 
    implode(', '$insert);
        if(!
    $result $admin->do_task($sql"convert_2db.php: Insert into table DEALS_SALT")) 
        { 
            echo 
    "error "$sql "<br />"
            
    $flag false
        } 
    As long as you don't run out of memory, it will create 1 insertion call for each file, instead of 1 insertion call for each line in each file.
    PHP Code:
    header('HTTP/1.1 420 Enhance Your Calm'); 
    Been gone for a few months, and haven't programmed in that long of a time. Meh, I'll wing it ;)

  • Users who have thanked Fou-Lu for this post:

    timgolding (01-11-2010)

  • #7
    Regular Coder ninnypants's Avatar
    Join Date
    Apr 2008
    Location
    Utah
    Posts
    504
    Thanks
    10
    Thanked 47 Times in 47 Posts
    Quote Originally Posted by Fou-Lu View Post
    No, 0 indicates no time limit imposed.
    Should have looked closer when I passed the parameters section thank you for correcting me

  • #8
    Senior Coder timgolding's Avatar
    Join Date
    Aug 2006
    Location
    Southampton
    Posts
    1,519
    Thanks
    114
    Thanked 110 Times in 109 Posts
    Quote Originally Posted by Fou-Lu View Post
    No, 0 indicates no time limit imposed.
    Try putting more weight against you're sql instead of you're php code. Develop you're insertions to take multiple inserts on a single execution:
    PHP Code:
        $insert = array();
        foreach(
    $data as $dat
        { 
                    
    //INSERT INTO tbl_name (a,b,c) VALUES(1,2,3,4,5,6,7,8,9); 
                    
    $insert[] = "('".mysql_real_escape_string($dat[0])."', '".mysql_real_escape_string($dat[1])."','".mysql_real_escape_string($dat[2])."','".mysql_real_escape_string($dat[3])."','".mysql_real_escape_string($dat[4])."','".mysql_real_escape_string($dat[5])."', '".mysql_real_escape_string($dat[6])."','".mysql_real_escape_string($dat[7])."','".mysql_real_escape_string($dat[8])."','".mysql_real_escape_string($dat[9])."','".mysql_real_escape_string($dat[10])."', '".mysql_real_escape_string($dat[11])."','".mysql_real_escape_string($dat[12])."','".mysql_real_escape_string($dat[13])."','".mysql_real_escape_string($dat[14])."','".mysql_real_escape_string($dat[15])."','".mysql_real_escape_string($dat[16])."')";
        }
        
    $sql "INSERT INTO `DEALS_SALT` (`SKU`,`HSNUM`,`TARIFFNUM`,`PRICE`,`FLR`,`HPLR`,`CASH_BACK`,`GIFT`,`RET`,`CASH_TYPE`,`CLEARANCE`, `STOCK`, `TOTAL_COST`, `EMLR`, `SKUPOPMONTH`, `DEALSRES1`, `DEALSRES2`) 
                    VALUES " 
    implode(', '$insert);
        if(!
    $result $admin->do_task($sql"convert_2db.php: Insert into table DEALS_SALT")) 
        { 
            echo 
    "error "$sql "<br />"
            
    $flag false
        } 
    As long as you don't run out of memory, it will create 1 insertion call for each file, instead of 1 insertion call for each line in each file.
    Thanks i tried your suggested code and echoed the mysql_error but i got this errors

    Code:
    DEALS_SALT table dropped
    DEALS_SALT table created
    error Got a packet bigger than 'max_allowed_packet' bytes
    error Got a packet bigger than 'max_allowed_packet' bytes
    error Got a packet bigger than 'max_allowed_packet' bytes
    error Got a packet bigger than 'max_allowed_packet' bytes
    error Got a packet bigger than 'max_allowed_packet' bytes
    error Got a packet bigger than 'max_allowed_packet' bytes
    error Got a packet bigger than 'max_allowed_packet' bytes
    error Got a packet bigger than 'max_allowed_packet' bytes
    error Got a packet bigger than 'max_allowed_packet' bytes
    error Got a packet bigger than 'max_allowed_packet' bytes
    error Got a packet bigger than 'max_allowed_packet' bytes
    Each file is 3.7MB unfortunately there is a lot of data to insert. Does anyone know the max allowed size of the packets?
    Last edited by timgolding; 01-11-2010 at 10:57 AM.
    You can not say you know how to do something, until you can teach it to someone else.

  • #9
    Senior Coder timgolding's Avatar
    Join Date
    Aug 2006
    Location
    Southampton
    Posts
    1,519
    Thanks
    114
    Thanked 110 Times in 109 Posts
    Ok i changed the max_allowed_packet to 8M and the script now runs untill it gets to the very bottom of part.8.txt . I looked at the data folder and there are part.9.txt , part.10.txt and part.11.txt still to process but the script seems to stop before processing these files. Any ideas why this might be?
    You can not say you know how to do something, until you can teach it to someone else.

  • #10
    Senior Coder timgolding's Avatar
    Join Date
    Aug 2006
    Location
    Southampton
    Posts
    1,519
    Thanks
    114
    Thanked 110 Times in 109 Posts
    I checked the file permissions on those files and found that part.8.txt permissions where set to 000 somehow. For some reason this was affecting the transition from part.8.txt to part.9.txt very strange behaviour. Anyway i finally got the script to finish running and display the results expected i will monitor it next few days and make sure it is completing.
    You can not say you know how to do something, until you can teach it to someone else.


  •  

    Tags for this Thread

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •