Just learned yesterday(today, haven’t slept yet) how to run a php script from the command line.
Had to because I was having trouble with the time limit when running my crawler(before I realized crawling a whole website without criteria was not going to work out).
I guess all those problems I had running my crawler locally(by starting firefox and running a page hosted locally in apache) by python/bash script would have vanished if I had just executed the page directly with php.
Just checked bing, there is no way to crawl their video search. If youtube closes their website to me(which they probably will) I hope it’s when my website is large enough to pay additional workers, though I do have a plan B(another video website, that may or may not be crawler-friendly)