Understanding the Synergy HTTP document transport API

The Synergy HTTP document transport API enables you to send and receive documents via HTTP or HTTPS from within your Synergy program.

This topic includes the following:

What is HTTP?

What is HTTPS?

What are OpenSSL and Schannel?

Using the HTTP document transport API

Transmitting data securely via HTTPS

Error messages

What is HTTP?

HTTP, which stands for HyperText Transfer Protocol, is a communication protocol for sending information via sockets. This information can be any type of data and is generically referred to as a “document.” As the underlying protocol for the web, HTTP defines how messages are formatted and transmitted and what actions servers and clients should take in response to commands.

What is HTTPS?

HTTPS, or secure HTTP, is an extension to HTTP that enables you to securely transmit sensitive data over a network to a server (most commonly a web server). It uses TLS (Transport Layer Security) as a sublayer under its regular HTTP application layering to encrypt and decrypt user requests as well as documents that are returned by the server. For example, if you do your banking online, the URL for the log-in page and any other account information you view will start with https://. If you submit a form to transfer funds, your browser’s HTTPS sublayer will encrypt it, the server will acknowledge the request on the same connection with an equally encrypted reply, and then your browser’s HTTPS sublayer will decrypt the acknowledgment for you.

HTTPS supports the use of X.509 digital certificates from the server so that, if necessary, a user can authenticate the sender. A digital certificate is like an electronic ID card that establishes your credentials when doing business or other transactions on the web. If you want to send encrypted messages, you can request a digital certificate from a certificate authority (CA). The CA digitally creates and issues a certificate that contains your name, a serial number, the expiration dates, a copy of the certificate holder’s public key (used for encrypting messages and digital signatures), and the digital signature of the certificate authority so that a recipient can verify the certificate is real. The recipient of an encrypted message uses the CA’s public key to decrypt the digital certificate attached to the message, verifies it as being issued by the CA, and gets the sender’s public key and other information from the certificate, which enables the recipient to send an encrypted reply. X.509 is the most widely used standard for defining digital certificates.

Important

Some countries are legally restricted from using HTTPS support, and it is illegal to export strong encryption to those countries. Refer to the Export Administration Regulations (EAR) of www.bis.doc.gov for current U.S. regulations.

Important

We recommend using TLS 1.2 or higher for secure sites. For more information, see Section 3.1 in the NIST publication, “Guidelines for the Selection, Configuration, and Use of Transport Layer Security (TLS) Implementations”. It’s very important that you maintain regular security updates to your OpenSSL software, the same as you do with your operating system.

What are OpenSSL and Schannel?

The Synergy HTTP document transport API interfaces with a third-party library—Schannel (Windows) or OpenSSL (Unix and OpenVMS)—to provide SSL support. Schannel (Microsoft Secure Channel) is included with the Windows operating system, but you may need to install OpenSSL on Unix and OpenVMS before you can use HTTPS.

Installing the OpenSSL libraries (Unix and OpenVMS)

For details on which version of OpenSSL is required for your operating system and where to get it, see OpenSSL requirements (Unix and OpenVMS). The OpenSSL libraries are installed in a standard location determined by the operating system. You don’t need to move them or set a path.

Once OpenSSL is installed, you can use the openssl command line utility to create a certificate request and key file, append a key file to the created certificate, establish a local CA for testing purposes, and more. Instructions for some of these tasks are provided in Using the HTTP document transport API and Transmitting data securely via HTTPS.

Additional documentation for the openssl utility can be found in the openssl manpages and at www.openssl.org/docs/.

Using the HTTP document transport API

Using the HTTP document transport API, a Synergy program, posing as an HTTP/HTTPS client, is able to send or receive data from an HTTP/HTTPS server. Conversely, a Synergy program can pose as an HTTP/HTTPS server to receive data from an HTTP/HTTPS client and then send data to that client to satisfy the client’s request.

This API uses the TCP/IP network protocol.

Important

You must include the synxml.def file, which defines some of the standard HTTP status codes and the text associated with these codes, to use the %HTTP_xxx routines.

URIs

Before you begin accessing or developing an HTTP or HTTPS server, you must know the URI (Uniform Resource Indicator) of the resource to be requested. A URI can have a maximum of 2000 bytes. URIs that are longer than 2000 bytes will be truncated.

The URI consists of

username:password@

Example: “bsmith:iq497j@”

Important

For security reasons, we do not recommend passing username:password as clear text in an HTTP packet.

If sending to a Synergy HTTP/HTTPS server, this can be a path or logical and filename. Examples:

“http://myServer:3507/DBLDIR:afile.txt”

“http://myName:myPassword@myServer:3507/c:/location/afile.txt”

If sending to a web server (such as IIS or Apache), location is a virtual directory set up on the web server, plus the page you want. Examples:

“http://myServer/location/file.jsp”

“http://myServer/location/file.asp”

For the client, you must build the URI with this information to send to the server. For the server, you must parse the specified URI to get the resource name and location.

The HTTP document transport API automatically escapes certain characters, such as spaces, in the URI when it sends a request to a server, and it also unescapes any escaped characters upon receiving a request from a client. (See the Escape Codes for Characters in a Memory Handle table for a list of characters that are escaped and their corresponding escape codes.) For example, “http://server/hello there.html” becomes “http://server/hello%20there.html”. To disable automatic escaping, set HTTP_NOESCAPE.

Note

The Synergy OPEN statement does not allow escape characters in a file specification. Therefore, if your program contains logic that uses part of the URI as the filename to save the handle data to, you may encounter problems on OPEN if the URI contains escape characters.

Memory handles

All memory that you allocate for the body of an HTTP/HTTPS transmission must be allocated with the DM_STATIC option of the %MEM_PROC routine. This memory must be global so it can be manipulated at different levels within the program. A memory allocation error occurs if a memory handle is not allocated with DM_STATIC.

All memory handle and length arguments (handle and length) are required arguments to the method calls. The memory handles are resized to the needed length by the %HTTP_xxx methods, and the correct length is returned.

In Synergy .NET, make sure that you explicitly close channels and free global memory handles. Implicit program or AppDomain shutdown does not ensure clean socket termination, and memory handles, channels, and sockets are not closed on AppDomain shutdown, only on application shutdown.

Proxy settings

The HTTP document transport API supports Synergy client HTTP and HTTPS requests through a proxy server. To specify the name and port of the proxy, use the PROXY_HOST and PROXY_PORT environment variables. If PROXY_HOST is not defined, the HTTP document transport API will not go through a proxy server.

You can change other default proxy settings using the PROXY_LOCAL and PROXY_SUBNET environment variables. If you’re having trouble connecting to local URIs, check the PROXY_SUBNET mask to ensure it matches your local subnet mask.

Accessing an HTTP server

(If you are accessing an HTTPS server, see Accessing an HTTPS server.)

To request a document from an HTTP server, your program should do the following:

1. Call %HTTP_CLIENT_GET, specifying the URI of the resource to get. An HTTP connection is established with the server.
2. After a response is received from the server (i.e., when %HTTP_CLIENT_GET returns), check the response status by checking the return value. If the return value is zero, the response body will contain the requested data. The HTTP connection will be closed. If the return value is nonzero, process the error.

To post a document to an HTTP server, follow the steps above using %HTTP_CLIENT_POST, providing the URI of the resource to post to and the data to be posted as the body of the request.

Developing an HTTP server

(If you are developing an HTTPS server, see Developing an HTTPS server.)

To pose as an HTTP server, your program should do the following:

1. Call %HTTP_SERVER_CREATE to create a server and initialize Synergy HTTP by connecting to the local host and a known port.
2. Wait to accept HTTP GET or POST requests using %HTTP_SERVER_RECEIVE. Inspect each request to find out the resource requested. Process the request and retrieve data as needed.

Your Synergy program prepares the HTTP response by setting the status value and putting the response data in the body of the HTTP response. (It puts the response text in the handle and returns the handle and its length as argument values.)

3. Call %HTTP_SERVER_SEND to send the HTTP response to the client.
4. Wait to accept another HTTP request.
5. When you’re finished, call %HTTP_SERVER_DELETE to release the memory associated with the server object.

Sample programs

See the URL_Encoding example available from Synergy CodeExchange in the Synergex Resource Center.

Transmitting data securely via HTTPS

If you are developing an HTTPS server, you will need a server certificate file. If you are accessing an HTTPS server, we highly recommend that you have a CA (Certificate Authority) file and, optionally, a certificate file (for the client).

A server certificate (file) contains public keys to prove identity between servers and clients. It binds an entity’s public key with the entity that owns the public/private key pair. A certificate is signed with the issuer’s private key so that anyone with the issuer’s public key can verify its authenticity.

A CA file essentially lists the certificate authorities whose certificates you are willing to trust. Thus, in order to make a connection to a secure server, it’s best practice to specify a CA file that contains the CA that issued the server’s certificate. If you don’t specify a CA file, the server certificate will not be verified, and you risk potential attack by someone attempting to spoof the server.

The easiest way to get a CA file is to export one from a known and reliable source, such as Microsoft Windows. Windows maintains information about all of the globally recognized Certificate Authorities, including VeriSign, Thawte, and many others. For details, see Exporting and importing certificates in Windows.

If the server you are trying to connect to uses a certificate that was not issued by a CA that is trusted by Windows (for example, if the server operator created their own certificate authority and used it to create and sign their own certificate), you will have to append information about that private CA to your CA file. The server operator will have to give you a copy of their CA’s public key, in PEM format, and you will have to manually add this certificate to your CA file by concatenating the two CA files (the one from Windows and the one from your server operator) together.

A client certificate (file) could also be involved when accessing an HTTPS server. A client certificate is a mechanism whereby the HTTPS server can verify who the client is. (This is rarely used, as normally only the client needs to verify who the server is.)

A client certificate is used to restrict which clients can connect to the server. If this is a requirement for the server, generally one of two things will happen. The most common is that the server operator will provide a certificate to the client and require that they pass this certificate on every request. (Note that this is not the same certificate as the “private” CA certificate discussed above. If the server operator is using a local certificate authority and requires client authentication, then two certificates will need to be provided: their CA certificate and a client certificate.) The other possibility is that the server operator will require the client operator to generate their own client certificate and to provide the public key of that certificate to them, so they can tell their server to trust it. This is very unlikely, however, as the server operator is likely to want to keep control of all certificates.

Accessing an HTTPS server

If accessing an HTTPS server that requires client certification, you will need to request a certificate from a trusted certificate authority. See Requesting a certificate from a certificate authority.

To request a document from an HTTPS server, your program should do the following:

1. (Optional) Call %HTTP_METHOD to register a Synergy routine that will get called to prompt the user for a password.
2. Call %HTTP_CLIENT_GET, specifying the URI of the resource to get, a CA filename, and optionally the client’s supported SSL protocols, supported ciphers, and certificate filename.

The client will authenticate the server certificate, and a secure connection will be established with the HTTPS server. If the supported protocols and ciphers are not specified, all protocols and ciphers will be accepted. The name of a client certificate file is only required when the client is requesting a resource on the server for which the server requires authentication.

3. After a response is received from the server (i.e., when %HTTP_CLIENT_GET returns), check the response status by checking the return value. If the return value is zero, the response body will contain the requested data. The secure connection will be closed. If the return value is nonzero, process the error.

To post a document to an HTTPS server, follow the steps above using %HTTP_CLIENT_POST, providing the URI of the resource to post to and the data to be posted as the body of the request.

Developing an HTTPS server

If developing an HTTPS server, you will need to request a certificate from a trusted certificate authority. See Requesting a certificate from a certificate authority.

Then, your program should do the following:

1. (Optional) Call %HTTP_METHOD to register a Synergy routine that will get called to prompt the user for a password.
2. Call %HTTP_SERVER_CREATE to create a server and initialize Synergy HTTPS by providing the name of a certificate file and optionally the server’s supported SSL protocols, supported ciphers, and CA filename.

The Synergy program running as an HTTPS server initializes by connecting to the local host and a known port. If no protocols or ciphers are specified, the default is TLS1.2. If a CA filename is specified, client authentication will be performed on all requests.

3. Wait to accept HTTPS GET or POST requests using %HTTP_SERVER_RECEIVE. Inspect each request to find out the resource requested. Process the request and retrieve data as needed.

Your Synergy program prepares the HTTPS response by setting the status value and putting the response data in the body of the HTTPS response. (It puts the response text in the handle and returns the handle and its length as argument values.)

4. Call %HTTP_SERVER_SEND to send the HTTPS response to the client.
5. Wait to accept another HTTPS request.
6. When you’re finished, call %HTTP_SERVER_DELETE to release the memory associated with the server object.

Ciphers (Unix and OpenVMS)

Several of the HTTP document transport API routines pass an optional ciphers argument, which specifies the encryption methods that are in effect in the form of a “cipher list.” (On Windows, this argument is ignored and the system default ciphers are used.) A cipher list consists of one or more cipher strings separated by colons. Commas or spaces are also acceptable separators, but colons are normally used.

The actual cipher string can take several different forms:

Each cipher string can be optionally preceded by the following characters:

Character

Meaning

!

The specified ciphers are permanently deleted from the list. The deleted ciphers can never reappear in the list even if they are explicitly stated.

-

The specified ciphers are deleted from the list, but some or all of the ciphers can be added again by later options.

+

The specified ciphers are moved to the end of the list. This option doesn’t add any new ciphers; it just moves matching existing ones.

None

The string is simply interpreted as a list of ciphers to be appended to the current preference list. If the list includes any ciphers already present, those ciphers will be ignored (i.e., they will not be moved to the end of the list).

Additionally the cipher string @STRENGTH can be used at any point to sort the current cipher list in order of encryption algorithm key length. When @STRENGTH is specified, HTTPS attempts to select the strongest cipher from the list.

For a current list of permitted cipher strings and their meanings, as well as lists of cipher suite names and their OpenSSL equivalents, see www.openssl.org/docs/manpages.html. Go to the “Commands” section for any version and select ciphers from the list of files.

Error messages

The following errors can occur when using the HTTP document transport API:

HTTP Document Transport API Errors

Message

Cause

Address in use

A socket error occurred when the server attempted to start on a port that was already being used.

Address not available

A socket error occurred. The requested address is not valid in this context.

CA file required

An HTTPS URI was passed to %HTTP_CLIENT_GET or %HTTP_CLIENT_POST, but the CA filename was not passed. A CA filename is required for an HTTPS request.

Cannot get certificate from file

The certificate file does not include the certificate section.

Cannot get private key from file

The certificate file does not include the private key section.

Cannot load random state

There is not enough random data to seed cryptographic algorithms. Define the HTTP_RAND environment variable to a file that the Synergy HTTP document transport API can use to gather random data.

Certificate verify failed

The certificate could not be validated using trusted CAs.

Connection refused

A socket error occurred when the client tried to connect to an invalid host and port.

Connection reset

A socket error occurred. If you are using HTTPS functionality, this could mean that only the client or only the server supports SSLv2 or that you tried to send an HTTP request to an HTTPS server. If you are using HTTP, the connection was closed by the remote host.

Error setting keepalive

A socket error occurred when the keepalive value was being set.

Error setting linger

A socket error occurred when the linger value was being set.

Host not found

A socket error occurred when an invalid host name was specified while the client was trying to connect to the server.

http request

The server expected an HTTPS request but received an HTTP request from the client.

HTTPS not available

HTTPS support is not available on the current platform. Either the OpenSSL library is missing or it is not the shared version.

Invalid http request

The server did not receive a well-formed HTTP request. The client may have tried to send an HTTPS request instead.
Invalid SSL protocol specified The SSL protocol specified in an HTTPS routine is less than the minimum required (currently SSLVER_TLS1_1, or TLS 1.1).

Invalid user information in URI

An invalid character was specified in the user information portion of the URI in a call to %HTTP_CLIENT_GET or %HTTP_CLIENT_POST. Valid characters are any alphanumeric character, escape codes %00 to %FF, semicolon (;), colon (:), ampersand (&), equal sign (=), plus sign (+), dollar sign ($), comma (,), hyphen (-), underscore (_), period (.), exclamation point (!), tilde (~), asterisk (*), single quotation mark ('), left parenthesis ( ( ), or right parenthesis ( ) ).

No certificate returned

The peer did not have a certificate.

No ciphers available

Either the ciphers on the client and server do not match or the cipher specified on the client is invalid.

No shared ciphers

The client and server do not support the same ciphers.

No such file or directory

The specified CA file or certificate filename does not exist.

Socket error number

The socket error number occurred. Socket error numbers are usually defined in Microsoft Developer Network (MSDN) help for Microsoft platforms and in the manpages or help pages of the particular socket function that failed for other platforms.

Socket is not connected

A socket error occurred. An attempt to send or receive data failed because the socket was not connected.

SSL handshake failure

This message usually appears on the client when there is a cipher mismatch or a certificate authorization issue. The most common reasons are as follows:

  • The server does not support the client’s SSL protocol version. Ensure the client is using a protocol version supported by the server.
  • The server and client do not share common cryptographic algorithms. Ensure the client is using a cipher supported by the server.
  • Server authentication failed. The server must have a valid certificate that can be traced to a CA that the client trusts.
  • Client authentication failed. The client must have a valid certificate that can be traced to a CA that the server trusts.

Timed out

A socket error occurred when the program timed out while waiting to accept or receive a request.

Unsupported protocol

The client and server protocols do not match.