Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 10 of 10
  1. #1
    New Coder
    Join Date
    May 2008
    Posts
    17
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Turn Recursive Function into FOR Loop: Solved

    Here is my code:

    Code removed...

    I want the function in red to be a loop instead of a recursive function so that I can cut down on memory use and the possible error of the stack being filled. My main problem is that a javascript for loop won't wait for an ajax request to complete before it continues executing code and that is what has kept me from writing a successful loop function. One you brilliant minds out there, can you give me a hand?
    Last edited by mberkom; 04-17-2009 at 06:50 PM. Reason: misspelling

  • #2
    Senior Coder rnd me's Avatar
    Join Date
    Jun 2007
    Location
    Urbana
    Posts
    4,371
    Thanks
    11
    Thanked 591 Times in 572 Posts
    you can use a sync request instead of the default async, that way the loop will wait for the data.
    my site (updated 13/9/26)
    BROWSER STATS [% share] (2014/9/03) IE7:0.1, IE8:4.6, IE11:9.1, IE9:3.1, IE10:3.0, FF:17.2, CH:46, SF:11.4, NON-MOUSE:38%

  • #3
    New Coder
    Join Date
    May 2008
    Posts
    17
    Thanks
    1
    Thanked 0 Times in 0 Posts
    And how would I do that? I think I understand what you are saying but does jquery have a sync request? Could you explain?

  • #4
    New Coder
    Join Date
    Mar 2009
    Location
    Fabric Covered Box
    Posts
    69
    Thanks
    1
    Thanked 16 Times in 14 Posts
    Does the scraped data have to be added to the page in the same order as the urls in the links array? If not, you could do something like
    Code:
    //			var i=0;
    			function parse_data(i){
    				$.post('/scraper/bot/get_page', { url: links[i]}, function(data){
    					if(state_sel != ''){var state = $(state_sel, data).html();}
    					for (var s=0; s <= ($(new_entry_sel, data).length-1); s++) {
    						//data gathered variables
    						if(person_name_sel != ''){var person_name = $(new_entry_sel, data).eq(s).find(person_name_sel).html();}
    						if(company_name_sel != ''){var company_name = $(new_entry_sel, data).eq(s).find(company_name_sel).html();}
    						if(email_sel != ''){var email = $(new_entry_sel, data).eq(s).find(email_sel).html();}
    						if(phone_sel != ''){var phone = $(new_entry_sel, data).eq(s).find(phone_sel).html();}
    						if(address_sel != ''){var address = $(new_entry_sel, data).eq(s).find(address_sel).html();}
    						if(link_sel != ''){var link = $(new_entry_sel, data).eq(s).find(link_sel).attr('href');}
    						if(custom_sel != ''){var custom = $(new_entry_sel, data).eq(s).find(custom_sel).html();}
    						
    						//insert data
    						$('#data').append('<tr><td>'+state+'</td><td>'+company_name+'</td><td>'+person_name+'</td><td>'+email+'</td><td>'+phone+'</td><td>'+address+'</td><td>'+link+'</td><td>'+custom+'</td></tr>');
    					};
    /*					if(i < (links.length-1)) {
    						i++;
    						return parse_data();
    					}
    					else {
    						alert('Finished');
    					}*/
    				});
    				
    			}
    			for(var i=0;i<links.length;i++)
                              parse_data(i);
    which would be faster due to the simultaneous requests, but the order wouldn't be guaranteed. Also, since I don't use jquery I'm not sure how well it handles simultaneous requests.

  • #5
    New Coder
    Join Date
    May 2008
    Posts
    17
    Thanks
    1
    Thanked 0 Times in 0 Posts
    This is what I've developed so far:

    Code:
    //loop through links, parse data, insert page
    			function parse_data(){
    				for (var i=0; i < (links.length - 1); i++) {
    					function ajax(){
    						$.ajax({
    							type: "POST",
    							url: "/scraper/bot/get_page",
    							data: "url="+links[i],
    							async: false,
    							success: function(data){
    								if(state_sel != ''){var state = $(state_sel, data).html();}
    								for (var s=0; s <= ($(new_entry_sel, data).length-1); s++) {
    									//data gathered variables
    									if(person_name_sel != ''){var person_name = $(new_entry_sel, data).eq(s).find(person_name_sel).html();}
    									if(company_name_sel != ''){var company_name = $(new_entry_sel, data).eq(s).find(company_name_sel).html();}
    									if(email_sel != ''){var email = $(new_entry_sel, data).eq(s).find(email_sel).html();}
    									if(phone_sel != ''){var phone = $(new_entry_sel, data).eq(s).find(phone_sel).html();}
    									if(address_sel != ''){var address = $(new_entry_sel, data).eq(s).find(address_sel).html();}
    									if(link_sel != ''){var link = $(new_entry_sel, data).eq(s).find(link_sel).attr('href');}
    									if(custom_sel != ''){var custom = $(new_entry_sel, data).eq(s).find(custom_sel).html();}
    									
    									//insert data
    									$('#data').append('<tr><td>'+state+'</td><td>'+company_name+'</td><td>'+person_name+'</td><td>'+email+'</td><td>'+phone+'</td><td>'+address+'</td><td>'+link+'</td><td>'+custom+'</td></tr>');
    								};
    							}
    						 });
    					}
    					ajax();	
    				};
    				alert('Finished');
    			}
    			parse_data();
    For some reason, I still get an error after about 25 entries put on the page that: script stack space quota is exhausted. I would like to figure out why that error occurs... And, if there is a solution.

  • #6
    New Coder
    Join Date
    Mar 2009
    Location
    Fabric Covered Box
    Posts
    69
    Thanks
    1
    Thanked 16 Times in 14 Posts
    "script stack space quota is exhausted" is a known bug: https://bugzilla.mozilla.org/show_bug.cgi?id=420869

  • #7
    New Coder
    Join Date
    May 2008
    Posts
    17
    Thanks
    1
    Thanked 0 Times in 0 Posts
    I realize it is a known bug, but how do I get around it? What in my code is triggering it?

  • #8
    New Coder
    Join Date
    Mar 2009
    Location
    Fabric Covered Box
    Posts
    69
    Thanks
    1
    Thanked 16 Times in 14 Posts
    Try this. By putting more the script into the global scope, and by using setTimeout to 'escape' from your current execution scope (allowing that old scope to be garbage collected), it should handle more/bigger files.
    Code:
    		<script type="text/javascript" charset="utf-8">
    			/*******************************************************
    			*Data Scraping Script                                  *
    			*******************************************************/
    			
    			//Global Vars
    			//------------------------------------------------------
    			
    			var session_id = <?=$session->id?>;
    			
    			//selectors
    			var new_entry_sel = '<?=$session->new_entry_selector?>';
    			var state_sel = '<?=$session->state_selector?>';
    			var person_name_sel = '<?=$session->person_name_selector?>';
    			var company_name_sel = '<?=$session->company_name_selector?>';
    			var email_sel = '<?=$session->email_selector?>';
    			var phone_sel = '<?=$session->phone_selector?>';
    			var address_sel = '<?=$session->address_selector?>';
    			var link_sel = '<?=$session->link_selector?>';
    			var custom_sel = '<?=$session->custom_selector?>';
    			
    			//array of links to be searched
    			var links = [
    <?php foreach($links->all as $l):?>
    '<?=trim($l->url)?>',
    <?php endforeach;?>
    ''
    			];
    			
    			
    			//recurse through links, parse data, print out onto page
    			function parse_data(){
    				$.post('/scraper/bot/get_page', { url: links[0]}, function(data){
    					    if(state_sel != ''){var state = $(state_sel, data).html();}
    					    for (var s=0; s <= ($(new_entry_sel, data).length-1); s++) {
    						    //data gathered variables
    						    if(person_name_sel != ''){var person_name = $(new_entry_sel, data).eq(s).find(person_name_sel).html();}
    						    if(company_name_sel != ''){var company_name = $(new_entry_sel, data).eq(s).find(company_name_sel).html();}
    						    if(email_sel != ''){var email = $(new_entry_sel, data).eq(s).find(email_sel).html();}
    						    if(phone_sel != ''){var phone = $(new_entry_sel, data).eq(s).find(phone_sel).html();}
    						    if(address_sel != ''){var address = $(new_entry_sel, data).eq(s).find(address_sel).html();}
    						    if(link_sel != ''){var link = $(new_entry_sel, data).eq(s).find(link_sel).attr('href');}
    						    if(custom_sel != ''){var custom = $(new_entry_sel, data).eq(s).find(custom_sel).html();}
    						
    						    //insert data
    						    $('#data').append('<tr><td>'+state+'</td><td>'+company_name+'</td><td>'+person_name+'</td><td>'+email+'</td><td>'+phone+'</td><td>'+address+'</td><td>'+link+'</td><td>'+custom+'</td></tr>');
    					      };
    					    links.shift();
    					    if(links.length) {
    						    setTimeout(parse_data,1);
    					      }
    					    else {
    						    alert('Finished');
    					      }
    				    });
    				
    			}
    		$(document).ready(function() {
    			parse_data();
    			
    		});
    		</script>

  • #9
    New Coder
    Join Date
    May 2008
    Posts
    17
    Thanks
    1
    Thanked 0 Times in 0 Posts
    No go... I get the same error.

  • #10
    New Coder
    Join Date
    Mar 2009
    Location
    Fabric Covered Box
    Posts
    69
    Thanks
    1
    Thanked 16 Times in 14 Posts
    Short of ditching jquery (most of these frameworks have a lot of overhead), I'm out of ideas.



  •  

    Tags for this Thread

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •