Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 6 of 6
  1. #1
    New to the CF scene
    Join Date
    Oct 2012
    Posts
    6
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Exclamation Downloading list of directories from website

    Hi there

    I need a piece of code that can download a list of directories from a given website. I know there's software such as WinHTTrack but that only downloads the files, not the subdirectories within a site.

    Is it possible to make a piece of code to download the names of all subdirectories from a website and save them to a log file?

    Thanks a lot.

  • #2
    Senior Coder rnd me's Avatar
    Join Date
    Jun 2007
    Location
    Urbana
    Posts
    4,380
    Thanks
    11
    Thanked 592 Times in 572 Posts
    look into the downthemall extension for firefox.

    you can use ajax to crawl the site and find all the linked-to folders, but to get a full list including hidden folders, you would need the co-operation of a server.
    my site (updated 13/9/26)
    BROWSER STATS [% share] (2014/9/03) IE7:0.1, IE8:4.6, IE11:9.1, IE9:3.1, IE10:3.0, FF:17.2, CH:46, SF:11.4, NON-MOUSE:38%

  • #3
    Supreme Master coder! Philip M's Avatar
    Join Date
    Jun 2002
    Location
    London, England
    Posts
    18,081
    Thanks
    203
    Thanked 2,542 Times in 2,520 Posts
    Quote Originally Posted by wman2 View Post
    Is it possible to make a piece of code to download the names of all subdirectories from a website and save them to a log file?
    Could I ask what legitimate reason do you have for wanting to do that?

    In any case JavaScript running in a browser is a client-side language. JavaScript does not have any commands for reading or writing files. Modern browsers can read files on the server using an Ajax call, but otherwise JavaScript has no capability to read from, write to, modify or delete a file (except a cookie), communicate with the server, access a database, the client's operating system or the Windows registry, or alter the default behaviour of the browser. And for security reasons JavaScript cannot access anything beyond the domain of the current page. This is known as the "same origin policy" and prevents a document or script loaded from one origin from getting or setting properties of a document from a different origin.
    Last edited by Philip M; 11-26-2012 at 07:55 AM.

    All the code given in this post has been tested and is intended to address the question asked.
    Unless stated otherwise it is not just a demonstration.

  • #4
    Senior Coder rnd me's Avatar
    Join Date
    Jun 2007
    Location
    Urbana
    Posts
    4,380
    Thanks
    11
    Thanked 592 Times in 572 Posts
    helpful guidance for lost souls:




    JavaScript running in a browser is a purely a client-side language. JavaScript has many commands for reading and writing files. Modern browsers can read files on the server using an Ajax/script call, and local files using an <input type=file> or drag and drop. JavaScript can write a specific file using the download attribute on <a> tags, or the FileWriter() interface where available, and can save any window.open()-based dataURL to a randomly-named file in the browser's download folder. JS communicates with a server using WebSockets, CORS, JSONP, importScripts(), new Image().src=url, and new EventSource(url). JavaScript can access a native database called indexedDB, interact with mySQL using php, and interact with the client's operating system using registerProtocolHander(), custom mimeTypes, plug-ins, and "Open With...". JavaScript largely ignores OS-specific features like the Windows registry, but it can alter the default behavior of the browser using opt-in permission requests for features like GPS, microphone, and camera access. JavaScript uses special interfaces to interact beyond the domain of the current page; CORS, JSONP, importScripts() and postMessage() are the most common.
    Last edited by rnd me; 11-26-2012 at 04:21 PM.
    my site (updated 13/9/26)
    BROWSER STATS [% share] (2014/9/03) IE7:0.1, IE8:4.6, IE11:9.1, IE9:3.1, IE10:3.0, FF:17.2, CH:46, SF:11.4, NON-MOUSE:38%

  • #5
    Supreme Master coder! Philip M's Avatar
    Join Date
    Jun 2002
    Location
    London, England
    Posts
    18,081
    Thanks
    203
    Thanked 2,542 Times in 2,520 Posts
    Quote Originally Posted by rnd me View Post
    helpful guidance for lost souls:




    JavaScript running in a browser is a purely a client-side language. JavaScript has many commands for reading and writing files. Modern browsers can read files on the server using an Ajax/script call, and local files using an <input type=file> or drag and drop. JavaScript can write a specific file using the download attribute on <a> tags, or the FileWriter() interface where available, and can save any window.open()-based dataURL to a randomly-named file in the browser's download folder. JS communicates with a server using WebSockets, CORS, JSONP, importScripts(), new Image().src=url, and new EventSource(url). JavaScript can access a native database called indexedDB, interact with mySQL using php, and interact with the client's operating system using registerProtocolHander(), custom mimeTypes, plug-ins, and "Open With...". JavaScript largely ignores OS-specific features like the Windows registry, but it can alter the default behavior of the browser using opt-in permission requests for features like GPS, microphone, and camera access. JavaScript uses special interfaces to interact beyond the domain of the current page; CORS, JSONP, importScripts() and postMessage() are the most common.
    If you say so. I have the idea that most of the items you mention are highly specialised and not available to normal Javascript client-side users. I suspect that some are only supported by certain browsers. In any case, they need the cooperation of both ends of the chain.

    All the code given in this post has been tested and is intended to address the question asked.
    Unless stated otherwise it is not just a demonstration.

  • #6
    Senior Coder rnd me's Avatar
    Join Date
    Jun 2007
    Location
    Urbana
    Posts
    4,380
    Thanks
    11
    Thanked 592 Times in 572 Posts
    Quote Originally Posted by Philip M View Post
    If you say so. I have the idea that most of the items you mention are highly specialised and not available to normal Javascript client-side users. I suspect that some are only supported by certain browsers. In any case, they need the cooperation of both ends of the chain.
    no, as a matter of fact, they are not special at all.

    The mentioned technologies work in current version all 5 major browsers (IE/FF/CH/OP/SA) with the following major exceptions:

    -eventSource() in IE.

    -navigator.registerProtocolHandler() is missing in IE, but as long as chrome or firefox is installed, the system-wide setting persists.

    -indexedDB is missing from safari + opera, but both of those support a polyfill using webSQL, so we use it without fear.

    -getUserMedia is missing in safari and IE, but safari has <input capture="camera"/>.

    getting specific, and referring to http://caniuse.com/, you will see surprising penetration.

    so, if we score 5 for full browser support and 4 for missing IE, and 1 if it only work in firefox, then with 20 items, 100 is full across-the-board support, and 20 would mean that each tech only works in one browser. 0 would imply no major browser runs any of the techs.



    Code:
    Cnt  Tech
    5  Ajax/script
    5  local files using an <input type=file>
    5  drag and drop files
    2  download attribute on <a>
    5  FileReader() 
    1  FileWriter() 
    5  dataURLs
    5  WebSockets
    5  CORS
    5  JSONP
    5  importScripts()
    5  new Image().src=url
    4  new EventSource(url)
    5  indexedDB
    4  registerProtocolHander()
    5  custom mimeTypes
    5  plug-ins, 
    5  "Open With..." (save/open dialog for unknown mimes)
    5  GPS,
    3  microphone
    4  camera
    5  postMessage()
    total: 88/100

    i knew i shouldn't have mentioned FileWriter(), but if i split dom scripts and ajax into 2 and throw away the FileWriter() (which i originally qualified as "where avail") then the score is 93/100. that's an A on any college campus.

    Most of the missing ones i mentioned are in IE, which is totally inapplicable to the fastest growing segment: tablets. Furthermore, IE 9 was the last monolithic IE; these tech will work their way into IE10 before long...

    Categorizing stuff that works in all major browsers more often than not, and mostly at least 'almost everywhere', as "highly specialised and not available to normal Javascript client-side users", is misleading at best and a disservice to noobs at worse.

    If one likes coding as though it's 2008, ok, that's certainly a viable option; but don't try to scare away folks who want to code like its 2012 by arming them with outdated information.
    Last edited by rnd me; 11-26-2012 at 10:05 PM.
    my site (updated 13/9/26)
    BROWSER STATS [% share] (2014/9/03) IE7:0.1, IE8:4.6, IE11:9.1, IE9:3.1, IE10:3.0, FF:17.2, CH:46, SF:11.4, NON-MOUSE:38%


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •