php - Receive "Internal Server Error" When Trying to Connect to FTP With Curl -
okay, kinda driving me nuts... keep receiving "internal server error" trying connect external ftp connection curl. can access ftp fine normally, not through curl.
i'm using codeigniter wrapper, don't think cause such issue. i've tried increasing memory/timeout, still can't in. 500 internal server error on page; can't figure out if curl returning anything, know normal error through curl (not internal server error) if disable 'username' or trying add 'password' (there no password ftp login).
here main scripts:
function ftpscrape() { parent::controller(); $this->load->model("curl_m"); } function index() { ini_set('memory_limit', '70000000m'); set_time_limit(0); define('cookies_dir', 'cookies'); define('random', cookies_dir . '/' . md5(uniqid(mt_rand(), true))); $this->curl_m->init(true, random . '.txt'); $this->curl_m->start(); $referer = ''; $url = 'ftp://ftp.server.com/'; $str = $this->curl_m->ftp($url, 'user', '', __line__, $referer); print "<br><br><textarea cols=\"80\" rows=\"20\">{$str}</textarea><br><br>"; $this->curl_m->close(); }
here "curl_m" model functions use:
function init($cookies = true, $cookie = 'cookies.txt', $compression = '', $proxy = '') { if(!is_dir('files')) { mkdir('files', 0777); } if(!is_dir('cookies')) { mkdir('cookies', 0777); } else if($dh = opendir('files/')) { while(($file = readdir($dh)) !== false) { if(is_file('files/'.$file)) { unlink('files/'.$file); } } } $this->user_agent = 'mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.9.1.7) gecko/20091221 firefox/3.5.7 (.net clr 3.5.30729)'; $this->compression = $compression; $this->proxy = $proxy; $this->cookies = $cookies; $this->filenumber = 0; $this->follow_location = 1; $this->headers_html = 0; $this->binary = 0; if($this->cookies == true) { $this->cookie($cookie); } } function start() { $this->process = curl_init(); } function close() { curl_close($this->process); } function ftp($url, $user = '', $pass = '', $line_no = '', $referer = '') { return $this->execute($url, '', $line_no, $referer, $user, $pass); } function execute($url, $data = '', $line_no = '', $referer = '', $user = '', $pass = '') { if(isset($this->headers)) { unset($this->headers); } if(preg_match('/\w/xsi', $data)) { $type = 'post'; } else { $type = 'get'; } $host = parse_url($url); $this->headers[] = 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'; $this->headers[] = 'accept-language: en-us,en;q=0.5'; $this->headers[] = 'accept-encoding: gzip,deflate'; $this->headers[] = 'accept-charset: iso-8859-1,utf-8;q=0.7,*;q=0.7'; $this->headers[] = 'keep-alive: 300'; $this->headers[] = 'connection: keep-alive'; $this->headers[] = 'expect:'; if(preg_match('/\w/xsi', $referer)) { $this->headers[] = 'referer: ' . $referer . ''; } if($type == 'post') { if(isset($this->content_type_change)) { $this->headers[] = 'content-type: ' . $this->content_type_change . ''; unset($this->content_type_change); } else { $this->headers[] = 'content-type: application/x-www-form-urlencoded'; } if(isset($this->extra_headings)) { $this->headers[] = $this->extra_headings; } if(!preg_match('/https:/xsi', $url)) { $this->headers[] = 'content-length: ' . strlen($data) . ''; } } curl_setopt($this->process, curlopt_url, $url); curl_setopt($this->process, curlopt_httpheader, $this->headers); curl_setopt($this->process, curlopt_header, $this->headers_html); curl_setopt($this->process, curlopt_useragent, $this->user_agent); if($this->cookies == true) { curl_setopt($this->process, curlopt_cookiefile, $this->cookie_file); } if($this->cookies == true) { curl_setopt($this->process, curlopt_cookiejar, $this->cookie_file); } curl_setopt($this->process, curlopt_encoding, $this->compression); curl_setopt($this->process, curlopt_connecttimeout, 0); curl_setopt($this->process, curlopt_timeout, 1200); curl_setopt($this->process, curlopt_ssl_verifypeer, 0); curl_setopt($this->process, curlopt_returntransfer, 1); if(strpos("ftp", $url) !== false) { curl_setopt($this->process, curlopt_ftplistonly, 1); } if(!empty($user)) { $usrpwd = $user . ':' . $pass; curl_setopt($this->process, curlopt_userpwd, $usrpwd); } if($this->binary == 1) { curl_setopt($this->process, curlopt_binarytransfer, 1); } if($this->follow_location == 1) { curl_setopt($this->process, curlopt_followlocation, 1); } else { curl_setopt($this->process, curlopt_followlocation, 0); } if($type == 'post') { curl_setopt($this->process, curlopt_postfields, $data); curl_setopt($this->process, curlopt_post, 1); } if(preg_match("/\w/", $this->proxy)) { curl_setopt($this->process, curlopt_proxy, $this->proxy); } $return = curl_exec($this->process); if($this->file_extension($url,$return) == 'jpg') { $fh = fopen('captcha.jpg', "w"); fwrite($fh, $return); fclose($fh); } unset($this->headers); return $return; }
does know why may having issue?
most of script created before started project (namely functions in curl_m model) converted class actual codeigniter model.
if can figure out how prevent causing internal server error should able fix rest enough.
well, figured out why wasn't working right. tried increase allowed memory , allow unlimited timeout, guess server didn't want allow me that.
after spent yesterday , today trying configure amazon ec2 account client set put of information on there, tested ftp scraper again , connected , retrieved data trying access. server testing, anyway--the final script going put on amazon ec2, putting off due complexity of setting (i'd never used amazon ec2 before).
either way, issue script either timing out or exceeding allotted memory. setting on higher-end server got connection work fine. apparently logging ftp curl takes more resources/time thought would.
Comments
Post a Comment