25.8 Using SSL

The SSL interface allows you to use Secure Socket Layer (SSL) with Lisp objects of type socket-stream and async-io-state.

The SSL interface is part of the "comm" module, so to load it you evaluate:

(require "comm")

Note: In this section we assume that the current package uses the comm package. That is, comm package symbols may not be qualified explicitly.

25.8.1 SSL implementations

The LispWorks SSL interface is implemented using an underlying SSL implementation, which may be either OpenSSL or the Apple Security Framework (sometimes shortened to just "Apple"). The Apple Security Framework implementation is new in LispWorks 8.0, and is available only on macOS 10.8 or later or iOS. It is the default implementation on these platforms. All other platforms and previous versions of LispWorks support only the OpenSSL implementation, so if you are not going to use SSL on Apple products all you need to know is to ignore any Apple specific features.

Implementation are named :openssl for OpenSSL and :apple for the Apple Security Framework.

In general, you will usually use only one of the implementations on a particular operating system, even on the operating systems that support both, but it is possible to use both of them at the same time (for different SSL connections). At any time, one of the implementations is the default implementation, and any SSL connections that are created without specifying the implementation explicitly will use this default implementation. To query and set the default implementation, you can use the accessor ssl-default-implementation. To check if an implementation is available, you can call the function ssl-implementation-available-p.

To make it easier to write code that can work with both implementations, as well as adding new features and simplifying using SSL, in LispWorks 8.0 and newer you can configure SSL connections using an abstract context. See 25.8.3 SSL abstract contexts for details.

For details of the underlying OpenSSL implementations, see the the OpenSSL documentation (often also available as man pages on Unix). For details of the Apple Security Framework, see the Security Framework documentation on the Apple developer site, and in particular the the Secure Transport section.

Detailed configuration of the SSL parameters can be done using the FLI, with OpenSSL or Apple Security Framework functions.

25.8.2 Obtaining and installing the OpenSSL library

At the time of writing, OpenSSL is available as shown in OpenSSL availability:

25.8.2.1 Installing the OpenSSL library on Solaris

After installing (with pkgadd) you need to put the shared libraries libcrypto.so and libssl.o on the loader path. By default these are installed in /usr/local/ssl/lib.

To add the libraries to the loader path, either:

• add /usr/local/ssl/lib to the environment variable LD_LIBRARY_PATH, or:
• create links from /usr/lib.

25.8.2.2 How LispWorks locates the OpenSSL libraries

Since OpenSSL is not a standard on all machines yet, the location of the library or libraries varies. By default, ensure-ssl loads libraries as shown in How LispWorks locates the OpenSSL libraries.

On machines where the path is unknown or is incorrect, you must set the path by calling set-ssl-library-path, or by passing the path as the library-path argument to ensure-ssl. The default setting for Windows matches the libraries from the page that is mentioned in the table OpenSSL availability.

25.8.3 SSL abstract contexts

SSL abstract contexts are objects that represent the configuration of SSL connections. They are created by using either create-ssl-server-context or create-ssl-client-context for creating server or client SSL connections respectively. They are then passed repeatably to functions that create socket connections (instances of socket-stream or async-io-state) or to functions that attach SSL to socket connections to configure the SSL, using the keyword :ssl-ctx.

See create-ssl-server-context and create-ssl-client-context for details about configuration options and their effects, and sections 25.8.4 Creating a stream with SSL and 25.8.5 Using Asynchronous I/O with SSL for the functions that take the :ssl-ctx argument.

Abstract contexts where introduced in LispWorks 8.0. They are intended to simplify code that needs to run on both SSL mplementation (see 25.8.1 SSL implementations), and simplify performing commonly executed tasks.

25.8.4 Creating a stream with SSL

There are four ways to make a socket-stream with SSL processing:

• Call create-ssl-socket-stream.
• Call (make-instance 'socket-stream :ssl-ctx ...).
• Call open-tcp-stream with the :ssl-ctx keyword.
• Call attach-ssl on a socket-stream.

When using the OpenSSL implementation, these calls implicitly load the OpenSSL library and seed the Pseudo Random Number Generator (PRNG). When using the Apple Security Framework implementation, they implictly load the Security Framework.

For example:

(open-tcp-stream some-url 443 :ssl-ctx t)

25.8.5 Using Asynchronous I/O with SSL

There are three ways to make an async-io-state with SSL processing:

• Call create-async-io-state-and-connected-tcp-socket with the :ssl-ctx keyword.
• Call accept-tcp-connections-creating-async-io-states with the :ssl-ctx keyword.
• Call async-io-state-attach-ssl on an async-io-state.

These calls implicitly load the OpenSSL library and seed the Pseudo Random Number Generator (PRNG).

25.8.6 Keyword arguments for use with SSL

The keyword arguments :ssl-ctx, :ssl-side, :ctx-configure-callback, :ssl-configure-callback and :handshake-timeout can be be passed to create and configure socket streams and async-io-states with SSL processing. However, in LispWorks 8.0 and newer, the preferred method of configuring SSL connections is to use 25.8.3 SSL abstract contexts with :ssl-ctx, in which case :ctx-configure-callback and :ssl-configure-callback are ignored, and :ssl-side is redundant. The various interface calls for creating and configuring SSL streams and async-io-states accept these keyword arguments as shown in SSL configuration keywords.

(make-instance 'socket-stream ...) and open-tcp-stream, when ssl-ctx is non-nil, call attach-ssl and pass it all the arguments. accept-tcp-connections-creating-async-io-states and create-async-io-state-and-connected-tcp-socket when ssl-ctx is non-nil attach the ssl similar to the way async-io-state-attach-ssl does.

:ssl-ctx specifies that SSL should be used, and also specifies its configuration. The value of ssl-ctx can be:

A symbol	Together with ssl-side, this symbol specifies which protocol to use. ssl-ctx can be one of: t or :default, meaning use the default. In LispWorks 8.0, that makes LispWorks select the latest version that is supported by the library. For OpenSSL 1.1, it will always be TLS (rather than SSL), up to :tls-v1-3. In earlier versions of OpenSSL, it will also accept :v32 if no TLS version is supported. For the Apple implementation, it specifies using :tls-v1-2 or later. Prior to LispWorks 8.0, t or :default meant the same as :v23. One of :tls-v1-3, :tls-v1-2, :tls-v1-1, :tls-v1, :v23, :v3 or :v2. In the OpenSSL implementation, the :v… keywords are mapped to the SSLv23_…, SSLv3_… and SSLv2_… methods, and the :tls-… keywords are mapped to TLS_… methods and also specify the minimum version of TLS to use. The underlying implementation (OpenSSL or Apple Security Framework) selects which version to use, which will be the highest that it can. An implementation name, which is one of :openssl or (in macOS or iOS) :apple. This forces use of the OpenSSL or Apple Security Framework SSL implementations respectively, but otherwise is like t described above. In OpenSSL implementation, LispWorks makes a new SSL_CTX object and uses it and frees it when the stream or state is closed. The interface calls also make an SSL object, uses it and frees it when the stream or state is closed. In the Apple implementation, LispWorks makes a new ssl-context-ref object, uses it and frees it when the stream or state is closed.
A ssl-abstract-context
	LispWorks creates implementation objects and configures them according to the specification in the ssl-abstract-context. See create-ssl-server-context and create-ssl-client-context for details. ssl-abstract-context was introduced in LispWorks 8.0, and we recommend that you use abstract contexts in all new code. Note that, even for the simplest case, when you can just pass t, reusing an abstract context is more efficient in OpenSSL (because it caches the SSL_CTX). Note that when a ssl-abstract-context is used, the keywords :ctx-configure-callback and :ssl-configure-callback are ignored, and :ssl-side is redundant.
A cons	Specifies a range of acceptable versions. The car of the cons must be a symbol as described in the symbol item above, and specifies the minimum acceptable version. The cdr must be one of of the :tls-v1-* symbols, and specifies the maximum acceptable protocol version. For example, to force use of TLS 1.2 use (:tls-v1-2 . :tls-v1-2).
A foreign pointer of type ssl-ctx-pointer (OpenSSL-specific)
	This corresponds to the C type SSL_CTX* in the OpenSSL implementation. This is used and is not freed when the stream is closed. The interface calls also make an SSL object, use it and free it when the stream is closed. The foreign pointer maybe a result of a call to make-ssl-ctx, but it can also be a result of your code, provided that it points to a valid SSL_CTX and has the type ssl-ctx-pointer.
A foreign pointer of type ssl-pointer (OpenSSL-specific)
	The referenced SSL is used and is not freed when the stream is closed. See the documentation for ssl-pointer for details.
A foreign pointer of type ssl-context-ref (Apple-specific)
	LispWorks takes ownership of the referenced SSL context and will release it when the stream is closed. See the documentation for ssl-context-ref for details.

When you pass a ssl-ctx-pointer or a ssl-pointer foreign pointer, these must have already been set up correctly and you are responsible for freeing them when they are no longer required.

:ssl-side specifies which side the stream is. When ssl-ctx is a ssl-abstract-context, :ssl-side is redundant, and if used must match the side of ssl-ctx. The value ssl-side can be one of :client, :server or :both (OpenSSL only). open-tcp-stream and create-async-io-state-and-connected-tcp-socket do not take this keyword and always use :client. For the other calls this argument defaults to :server.

In the OpenSSL implementation, the value of ssl-side is used in three cases:

• When a new SSL_CTX object is created, it is used to select the method:

  :client => …_client_method

  :server => …_server_method

  :both => …_method

• When ssl-ctx is of type ssl-ctx-pointer, it checks that the side of ssl-ctx and ssl-side are not conflicting. If one is :client and the other is :server, they conflict and an error is signaled.
• When a new SSL object is created, when ssl-side is either :client or :server, LispWorks calls ssl-set-connect-state or ssl-set-accept-state respectively.

In the Apple implementation, ssl-side is used to select the protocol side in the call to SSLCreateContext when creating a new ssl-context-ref.

In the OpenSSL implementation, when a new SSL object is created, ssl-side is :client and handshake-timeout is greater than 0, a handshake is performed immediately.

In the Apple implementation, a handshake is always performed immediately after attaching SSL to a socket.

If ssl-ctx is of type ssl-pointer or ssl-context-ref then ssl-side is ignored.

:ctx-configure-callback specifies an OpenSSL-specific callback, a function which takes a foreign pointer of type ssl-ctx-pointer. This is called immediately after a new SSL_CTX is created. If the value of ssl-ctx is not a symbol, ctx-configure-callback is ignored.

:ssl-configure-callback specifies a callback, a function which takes a foreign pointer of type ssl-pointer or ssl-context-ref. This is called immediately after a new ssl-pointer or ssl-context-ref is created. If the value of ssl-ctx is a ssl-pointer, ssl-context-ref or ssl-abstract-context ssl-configure-callback is ignored. ote that abstract contexts have separate callbacks for the different implementations, and therefore it is much more convenient to use abstract contexts in code that needs this callback and is intended to be used on more than one implementation.

When a handshake is performed immediately (in the Apple implementation or in the OpenSSL implementation when ssl-side is :client and ssl-ctx is not a ssl-pointer), handshake-timeout specifies the time in seconds to wait for the handshake to complete. If handshake-timeout is nil (the default) then it waits indefinitely, but the underlying implementation may have its own timeout which will cause a failure after a while. If the handshake fails or times out, it is an error situation, and an error is signaled as described in 25.8.8 Errors in SSL.

In typical usage, you will create few ssl-abstract-context objects (maybe only one), configure them as appropriate for your application and the machine that it runs on, and then use one of these as ssl-ctx in all of your calls. If some connections need special configuration, you will use ssl-configure-callback in the ssl-abstract-context to configure the SSL of this connection. Sometimes when you open a connection as a client it may be sufficient to pass a symbol for ssl-ctx. Passing an ssl-pointer or ssl-context-ref as ssl-ctx is for special cases.

25.8.7 Attaching SSL to an existing socket

You can attach SSL to an existing socket-stream by calling attach-ssl on the stream. The socket-stream SSL keyword arguments are processed by attach-ssl as described in 25.8.6 Keyword arguments for use with SSL.

Detach SSL from a socket-stream and shut down the SSL with detach-ssl.

For full descriptions see attach-ssl and detach-ssl.

You can attach SSL to an existing async-io-state by calling async-io-state-attach-ssl on the state, and detach it using async-io-state-detach-ssl.

Notes:

After an object (stream or state) has been detached, you can attach SSL to it again.

Detaching frees any automatically generated SSL objects in the same way that closing a stream or state does.

The SSL objects are attached to the socket-stream or async-io-state, rather that to the socket. Therefore if you want to move a socket to another object then you need to attach it again.

For example, if you have attached SSL to an async-io-state and then want to change to synchronous communication, you need to close the async-io-state by close-async-io-state with keep-alive true (effectively detach the SSL), and then call make-instance with socket-stream with the socket plus SSL-CTX and any other necessary arguments.

To move the other way, from synchronous to asynchronous, use replace-socket-stream-socket with socket nil to disconnect the socket from the stream (which effectively calls detach-ssl), call create-async-io-state with the socket, and then call async-io-state-attach-ssl on the new async-io-state.

25.8.8 Errors in SSL

If there are errors inside SSL, LispWorks will signal an error of type ssl-condition, which is a subclass of socket-error.

The condition can be one of the types ssl-closed, ssl-error, ssl-failure, ssl-handshake-timeout, ssl-verification-failure and ssl-x509-lookup See the manual pages for details of these condition classes.

The exact meaning of signaling a SSL error depends on the context. For synchronous socket I/O (using socket-stream), it means calling error, except when it happens inside a function that takes errorp argument (open-tcp-stream and create-ssl-socket-stream) and errorp is nil. In the latter case, these functions return the condition object as a second value.

For asynchronous socket I/O (using async-io-state), the condition will be part of the format arguments list that is passed to callback in create-async-io-state-and-connected-tcp-socket or async-io-state-attach-ssl. You can get the condition from this list by using async-io-ssl-failure-indicator-from-failure-args. In accept-tcp-connections-creating-async-io-states, the condition will be the argument of ssl-error-callback.

25.8.9 Examples of using the socket stream SSL interface

See the example files in:

(example-edit-file "ssl/")