Creating a drop down list in html. Tree-like lists of different variations
c URL is a very useful command line tool for transferring data from or to a server. Curl supports various protocols such as FILE, HTTP, HTTPS, IMAP, IMAPS, LDAP, DICT, LDAPS, TELNET, FTPS, GOPHER, RTMP, RTSP, SCP, SFTP, POP3, POP3S, SMB, SMBS, SMTP, SMTPS, and TFTP.
cURL can be used in many different and interesting ways. With this tool you can download, upload and manage files, check your address Email, or even update your status on some social media websites or check the weather outside. In this article, we will look at five of the most useful and basic uses of the cURL tool on any .
1. Check URL
One of the most common and simplest uses of cURL is printing the command itself followed by the URL you want to test
Curl https://domain.ru
This command will display the contents of the URL on your terminal
2. Save URL output to a file
Curl -o website https://domain.ru % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 41793 0 41793 0 0 275k 0 --:--:-- --:--:-- --:--:-- 2.9M
In this example, the output will be saved to a file named 'website' in the current working directory.
3. Download Files with Curl
You can download files with Curl by adding the -o option to the command. It is used to save files to local server with the same names as on the remote server
Curl -O https://domain.ru/file.zip
In this example, the 'file.zip' archive will be downloaded to the current working directory.
You can also upload a file with a different name by adding the -o option to cURL.
Curl -o archive.zip https://domain.ru/file.zip
So the 'file.zip' archive will be downloaded and saved as 'Archive.zip'.
cURL can also be used to download multiple files at the same time, as shown in the example below
Curl -O https://domain.ru/file.zip -O https://domain.com/file2.zip
Curl can also be used to download files securely over SSH with the following command
Curl -u user sftp://server.domain.ru/path/to/file
Please note that you must use full path to the file you want to download
4. Get information from the website's HTTP header
You can easily get HTTP header information from any website by adding the -I ('i') option to cURL.
Curl -I http://domain.ru HTTP/1.1 200 OK Date: Sun, 16 Oct 2016 23:37:15 GMT Server: Apache/2.4.23 (Unix) X-Powered-By: PHP/5.6.24 Connection: close Content-Type: text/html; charset=UTF-8
5. FTP Server Access
To access the FTP server using Curl, you need to use the following command
Curl ftp://ftp.domain.ru --user username:password
Curl will connect to the FTP server and list all files and directories in the user's home directory
You can download the file using FTP
Curl ftp://ftp.domain.ru/file.zip --user username:password
and upload the file to the FTP server
Curl -T file.zip ftp://ftp.domain.ru/ --user username:password
You can check the Curl page manually to see all available cURL options and its functionality
man-curl
PS. If you liked this post, please share it with your friends in social networks using the buttons below or just leave a comment. Thank you.
This article assumes that you are familiar with the basics of networking and the HTML language.
The ability to write scripts is essential in building a good computer system. The extensibility of Unix systems through shell scripts and various programs that execute automated commands is one of the reasons why they are so successful.
The increasing number of applications that are moving to the web has led to the fact that the topic of HTTP scripts is becoming more and more in demand. Important tasks in this area are the automatic extraction of information from the Internet, sending or downloading data to web servers, etc.
Curl is a command line tool that allows you to do URL manipulation and passing various kinds. This article focuses on making simple HTTP requests. It is assumed that you already know where to dial
# curl --help
# curl --manual
for information about curl.
Curl is not a tool that will do everything for you. It creates requests, receives data, and sends data. You may need some "glue" to hold everything together, perhaps some scripting language (like bash) or a few manual calls.
1. HTTP protocol
HTTP is the protocol used when receiving data from web servers. It is a very simple protocol that is built on top of TCP/IP. The protocol also allows information to be sent to the server from the client using several methods, as will be shown next.
HTTP are strings of ASCII text sent from a client to a server to request some action. When a request is received, the server responds to the client with several service text lines, and then with the actual content.
Using the curl -v option, you can see which commands curl sends to the server, as well as other informational text. The -v switch is perhaps the only way to debug or even understand the interaction between curl and the web server.
2.URL
The URL format (Uniform Resource Locator - universal resource address) specifies the address of a specific resource on the Internet. You probably know this, examples of URLs are http://curl.haxx.se or https://yourbank.com.
3. Get (GET) page
The simplest and most common HTTP request is to get the content of a URL. The URL can refer to a web page, an image, or a file. The client sends a GET request to the server and receives the requested document. If you run the command
# curl http://curl.haxx.se
you will get a web page displayed in your terminal window. The complete HTML document contained at this URL.
All HTTP responses contain a set of headers that are usually hidden. To see them along with the document itself, use the curl -i option. You can also request only headers with the -I switch (which will force curl to make a HEAD request).
4. Shapes
Forms are the primary way of presenting a website as an HTML page with fields in which the user enters data and then clicks an OK or Submit button, after which the data is sent to the server. The server then uses the received data and decides how to proceed: look up the information in the database, show the entered address on the map, add an error message, or use the information to authenticate the user. Of course, there is some program on the server side that accepts your data.
4.1 GET
The GET form uses the GET method, like this:
If you open this code in your browser, you will see a form with a text box and a button that says "OK". If you enter "1905" and click OK, the browser will generate a new URL to follow. The URL will be a string consisting of the path of the previous URL and a string like "junk.cgi?birthyear=1905&press=OK".
For example, if the form was located at "www.hotmail.com/when/birth.html", then clicking the OK button will take you to the URL "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK".
Majority search engines work this way.
To have curl generate a GET request, simply enter what you would expect from the form:
# curl "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK"
4.2 POST
The GET method causes all the entered information to be displayed in the address bar of your browser. This may be fine when you need to bookmark a page, but it's an obvious disadvantage when you're entering secret information into form fields, or when the amount of information entered into the fields is too large (resulting in an unreadable URL).
The HTTP protocol provides the POST method. With it, the client sends data separately from the URL and therefore you will not see it in the address bar.
The form that generates the POST request is similar to the previous one:
Curl can form a POST request with the same data as follows:
# curl -d "birthyear=1905&press=%20OK%20" www.hotmail.com/when/junk.cgi
This POST request uses "Content-Type application/x-www-form-urlencoded", which is the most widely used way.
The data you send to the server must be properly encoded, curl won't do it for you. For example, if you want the data to contain a space, you need to replace that space with %20, and so on. Lack of attention to this issue is a common mistake, due to which the data is not transmitted as it should.
Back in 1995 it was determined additional way transfer data over HTTP. It is documented in RFC 1867, which is why it is sometimes referred to as RFC1867-posting.
This method is mainly designed to better support file uploads. The form that allows the user to upload a file looks like this in HTML:
Note that the Content-Type is set to multipart/form-data.
To send data to such a form using curl, enter the command:
# curl -F [email protected]-Fpress=OK
4.4 Hidden fields
A common way to communicate state information in HTML applications is by using hidden fields in forms. Hidden fields are not filled in, they are invisible to the user and are passed in the same way as normal fields.
A simple example of a form with one visible field, one hidden field and an OK button:
To send a POST request with curl, you don't have to think about whether the field is hidden or not. For curl they are all the same:
# curl -d "birthyear=1905&press=OK&person=daniel"
4.5 Find out what a POST request looks like
When you want to fill out a form and send data to the server using curl, you probably want the POST request to look exactly like the one made using the browser.
An easy way to see your POST request is to save the form's HTML page to disk, change the method to GET, and hit the "Submit" button (you can also change the URL to which the data will be submitted).
You will see that the data has been appended to the URL, separated by "?" characters, as expected when using GET forms.
5. PUT
Perhaps, The best way upload data to HTTP server is to use PUT. Again, this requires a program (script) on the back end that knows what to do and how to accept an HTTP PUT stream.
Send a file to the server using curl:
# curl -T uploadfile www.uploadhttp.com/receive.cgi
6. Authentication
Authentication - passing a username and password to the server, after which it checks if you have the right to perform the requested request. Basic authentication (which curl uses by default) is clear text authentication, which means that the username and password will not be encrypted, but only slightly "fogged" with the Base64 algorithm, which leaves it possible for attackers to find out this information on the way between you and the HTTP server.
Tell curl to use username and password:
# curl -u name:password www.secrets.com
The site may require the use of a different authentication method (see what the server writes in the headers), in these cases, you can use the --ntlm, --digest, --negotiate or even --anyauth keys. Sometimes access to external HTTP servers occurs through a proxy, as is often done in companies and firms. An HTTP proxy may require its own username and password to access the Internet. Relevant curl key:
# curl -U proxyuser:proxypassword curl.haxx.se
If the proxy requires NTLM authentication, specify --proxy-ntlm, if the Digest method, then --proxy-digest.
If you do not specify a password in the -u and -U options, then curl will ask you for it interactively.
Note that when curl is running, the run string (and with it the keys and passwords) may be visible to other users on your system in the task list. There are ways to prevent this. More on that below.
7. Referer
An HTTP request may include a "referer" field that indicates the URL from which the user came to this resource. Some programs/scripts check the "referer" field and do not execute the request if the user came from an unknown page. Although this is a silly way to check, many scripts use it nonetheless. With curl, you can put anything in the "referer" field and thus force it to do what you want.
This is done in the following way:
# curl -e http://curl.haxx.se daniel.haxx.se
8. User agent
All HTTP requests support a "User-Agent" field that specifies the user's client application. Many web applications use this information to display the page in one way or another. Web programmers create multiple versions of a page for users different browsers to improve appearance, using various javascript scripts, vbscript, etc.
Sometimes you may find that curl returns a page that is not what you saw in your browser. In this case, it is just appropriate to use the "User Agent" field in order to once again deceive the server.
Disguise curl as Internet Explorer on a Windows 2000 machine:
# curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
Why not become Netscape 4.73 on a Linux machine (PIII):
# curl -A "Mozilla/4.73 (X11; U; Linux 2.2.15 i686)"
9. Redirects
In response to your request, the server, instead of the page itself, may return an indication of where the browser should go next to get to desired page. The header that tells the browser this redirect is "Location:".
By default, curl does not go to the address specified in "Location:", but simply displays the page as usual. But you can send it like this:
# curl -L www.sitethatredirects.com
If you're using curl to POST requests to a site that immediately redirects to another page, you can safely use -L and -d/-F. Curl will make a POST request for the first page and then a GET request for the next one.
10. Cookies
Cookies allow web browsers to control state on the client side. Cookie is a name with content attached. The server, by sending cookies, tells the client the path and host name where the next time cookies should be sent, tells the cookie lifetime and some other parameters.
When a client connects to the server at the address specified in the received cookie, the client sends that cookie to the server (if the lifetime has not expired).
Many applications and servers use this method to combine multiple requests into one logical session. In order for curl to also perform this function, we must be able to save and send cookies, just like browsers do.
The simplest way to send a cookie to the server when fetching a page with curl is to add the appropriate key to command line:
# curl -b "name=Daniel" www.cookiesite.com
Cookies are sent as regular HTTP headers. This allows curl to store cookies by storing headers. Saving cookies with curl is done with the command:
# curl -D headers_and_cookies www.cookiesite.com
(by the way, it is better to use the -c switch to save cookies, more on that below).
curl has a fully featured cookie handler which is useful when you want to connect to the server again and use the cookies you saved last time (or hand-crafted). To use cookies stored in a file, call curl like this:
# curl -b stored_cookies_in_file www.cookiesite.com
The curl cookie engine is enabled when you specify the -b switch. If you want curl to only accept cookies, use -b with a file that doesn't exist. For example, if you want curl to accept cookies from a page and then follow a redirect (perhaps giving away the cookie that was just accepted), you can call curl like this:
# curl -b nada -L www.cookiesite.com
Curl can read and write cookies in Netscape and Mozilla format. This convenient way exchange cookies between browsers and automatic scripts. The -b switch automatically determines whether given file cookies of the specified browsers and handles it accordingly, and using the -c/--cookie-jar option you can force curl to write a new cookie when the operation completes:
# curl -b cookies.txt -c newcookies.txt www.cookiesite.com
11. HTTPS
There are several ways to secure your HTTP transmissions. The most well-known protocol that solves this problem is HTTPS, or HTTP over SSL. SSL encrypts all data sent and received over the network, which increases the likelihood that your information will remain secret.
Curl supports requests to HTTPS servers thanks to the free OpenSSL library. Requests occur in the usual way:
# curl https://that.secure.server.com
11.1 Certificates
In the HTTPS world, you use certificates for authentication in addition to the username and password. Curl supports certificates on the client side. All certificates are locked with a passphrase that you need to enter before curl can start working with them. Key phrase can be specified either on the command line or entered interactively. Certificates in curl are used like this:
# curl -E mycert.pem https://that.secure.server.com
Curl also authenticates the server by verifying the server's certificate against a locally stored one. A mismatch will result in curl refusing to connect. To ignore authentication, use the -k switch.
More detailed information certificates can be found at http://curl.haxx.se/docs/sslcerts.html.
12. Arbitrary request headers
You may need to modify or add elements of individual curl requests.
For example, you can change POST request on PROPFIND and send the data as "Content-Type: text/xml" (instead of the usual Content-Type):
# curl -d "
You can remove any header by specifying it without content. For example, you can remove the "Host:" header, thereby making the request "empty":
# curl -H "Host:" http://mysite.com
You can also add headers. Your server may need a "Destination:" header:
# curl -H "Destination: http://moo.com/nowhere" http://url.com
13. Debugging
It often happens that a site responds to curl requests differently than browser requests. In this case, you need to assimilate curl to the browser as much as possible:
- Use the --trace-ascii switch to save a detailed report of requests so that you can examine them in detail and understand the problem.
- Make sure you check for cookies and use them when needed (read -b switch and save -c switch)
- Specify one of the latest popular browsers in the "user-agent" field
- Fill in the "referer" field as the browser does
- If you are using POST requests, make sure that all fields are passed in the same order as the browser (see above, point 4.5)
A good helper in this difficult task is the Mozilla/Firefox LiveHTTPHeader plugin, which allows you to view all the headers that this browser sends and receives (even when using HTTPS).
A more low level approach is to capture HTTP traffic on the network using programs like ethereal or tcpdump and then analyze what headers were received and sent by the browser (HTTPS makes this approach inefficient).
RFC 2616 is required reading for anyone who wants to understand the HTTP protocol.
RFC 2396 explains the URL syntax.
RFC 2109 defines how cookies work.
RFC 1867 defines the File Upload Post format.
http://openssl.planetmirror.com - Homepage the OpenSSL project
http://curl.haxx.se - cURL project homepage
A real practical example: you need to reboot the router (modem) to change the IP address. To do this, you need to: log in to the router, go to the maintenance page and click the "Reboot" button. If this action needs to be performed several times, then the procedure must be repeated. Agree, you don’t want to do this routine manually every time. cURL allows you to automate all of this. With just a few cURL commands, you can achieve authorization and complete the task on the router.
- cURL is handy for getting data from websites on the command line.
Those. cURL use cases are quite real, although, in most cases, programmers who use it for their programs need cURL.
CURL supports many protocols and authorization methods, can transfer files, works correctly with cookies, supports SSL certificates, proxies, and much more.
cURL in PHP and command line
We can use cURL in two main ways: in PHP scripts and on the command line.
To enable cURL in PHP on the server, you need to uncomment the line in the php.ini file
Extension=php_curl.dll
And then restart the server.
On Linux, you need to install the curl package.
On Debian, Ubuntu or Linux Mint:
$ sudo apt-get install curl
On Fedora, CentOS or RHEL:
$ sudo yum install curl
To clearly see the difference in usage in PHP and on the command line, we will perform the same tasks twice: first in PHP script and then on the command line. Let's try not to get confused.
Getting data with cURL
Getting data with cURL in PHP
PHP example:
Everything is very simple:
$target_url- the address of the site that we are interested in. After the site address, you can put a colon and add the port address (if the port is different from the standard one).
curl_init- initializes a new session and returns a handle, which in our example is assigned to a variable $ch.
We then execute the cURL request with the function curl_exec, which is passed a descriptor as a parameter.
Everything is very logical, but when this script is executed, the content of the site will be displayed on our page. But what if we do not want to display the content, but want to write it to a variable (for further processing or parsing).
Let's add a little to our script:
0) ( echo "curl error: " . curl_error($ch); ) curl_close($ch); ?>
We have a line curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);.
curl_setopt- sets options. Full list options can be found on this page:
Hidden from guests
$response_data = curl_exec($ch);
Now the value of the script is assigned to the $response_data variable, which can be used for further operations. For example, you can display its contents.
Stitches
If (curl_errno($ch) > 0) ( echo "curl error: " . curl_error($ch); )
are used for debugging in case of errors.
Getting data with cURL on the command line
On the command line, just type
where instead of mi-al.ru- the address of your site.
If you need to copy the data to a variable, and not display the result on the screen, then do this:
Temp="curl mi-al.ru"
However, some data is still displayed:
So that they are not displayed, add the key -s:
Temp="curl -s mi-al.ru"
You can see what has been recorded:
echo $temp | less
Basic and HTTP authenticationAuthentication, simply put, is the introduction of a username and password.
Basic authentication is server authentication. For this, two files are created: .htaccess And .htpasswd
The content of the .htaccess file is something like this
AuthName "Only for registered users!" AuthType Basic require valid-user AuthUserFile /home/freeforum.biz/htdocs/.htpassw
The content of the .htpasswd file is something like this:
Mial:CRdiI.ZrZQRRc
Those. login and password hash.
When you try to access a password-protected folder, the browser will display something like this:
HTTP authentication is the case when we enter a username and password into a form on a website. It is this authentication that is used when entering mail, forums, etc.
cURL Basic Authentication (PHP)
There is a website
Hidden from guests
Let's try our initial script:
0) ( echo "curl error: " . curl_error($ch); ) else ( echo $response_data; ) curl_close($ch); ?>
Although the script considers that there is no error, we do not like the output result at all:
We add two lines:
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC); curl_setopt($ch, CURLOPT_USERPWD, "ru-board:ru-board");
In the first line we set the type of authentication - basic. The second line contains the name and password separated by a colon (in our case, the name and password are the same - ru-board). It turned out like this:
0) ( echo "curl error: " . curl_error($ch); ) else ( echo $response_data; ) curl_close($ch); ?> Trying: 30946 Great! Basic cURL authentication (on the command line) The same can be achieved on the command line with one line: curl -u ru-board:ru-board http://62.113.208.29/Update_FED_DAYS/
I didn't forget to specify the authentication type, it's just that in cURL the basic authentication type is the default.
On the command line, everything turned out so quickly that, out of frustration, I wrote this program. She connects to the site and downloads the latest update:
Temp=`curl -s -u ru-board:ru-board http://62.113.208.29/Update_FED_DAYS/ | grep -E -o "Update_FED_201(1).(2).(2).7z" | uniq | tail -n 1`; curl -o $temp -u ru-board:ru-board http://62.113.208.29/Update_FED_DAYS/$temp
Just a few more commands can be added:
- unpacking the archive to the specified directory;
- launch of ConsultantPlus updates (these are updates for him);
- you can implement a check - whether the last available update has already been downloaded or a new one has appeared;
- add it all to Cron for daily updates.
cURL HTTP Authentication in PHP
We need to know:
- address where to send data for authentication
- send method GET or POST
- login
- password
The address where you want to send data can be taken from the authentication form. For example: