h2spec is an excellent test tool to check if HTTP/2 servers can handle error cases correctly. When I was developing HTTP/2 server library in Haskell, I used to utilize Firefox and Chrome for normal cases and h2spec for error cases. h2spec much helped me to improve the quality of the library. What is surprised is that the author, Moto Ishizawa, is still enhancing h2spec!
Since the QUIC library in Haskell supports both the client side and the server side, normal cases are tested by itself. I wanted to test error cases, too. After considering for two weeks, it appeared that QUIC error cases can be made easily. The key idea is providing hooks to covert data structures:
data Hooks = Hooks { onPlainCreated :: EncryptionLevel -> Plain -> Plain , onTransportParametersCreated :: Parameters -> Parameters , onTLSExtensionCreated :: [ExtensionRaw] -> [ExtensionRaw] }
If we want to test an error case where a unknown frame is included a QUIC packet, we can provide the following function for onPlainCreated:
unknownFrame :: EncryptionLevel -> Plain -> Plain unknownFrame lvl plain | lvl == RTT1Level = plain { plainFrames = UnknownFrame 0x20 : plainFrames plain } | otherwise = plain
Hooks is passed to the client/server runners via configuration. I used our lovely hspec to run the test:
transportSpec :: ClientConfig -> SpecWith a transportSpec cc0 = do describe "QUIC servers" $ do it "MUST send FRAME_ENCODING_ERROR if a frame of unknown type is received [Transport 12.4]" $ \_ -> do let cc = addHook cc0 $ setOnPlainCreated unknownFrame runC cc waitEstablished `shouldThrow` transportError FrameEncodingError transportError :: TransportError -> QUICError -> Bool transportError te (TransportErrorOccurs te' _) = te == te' transportError _ _ = False
The notation of `shouldThrow` transportError FrameEncodingError is really cool, isn't it?
After adding some error cases, I hit upon an idea of a command line tool with this error cases reused by adding another main function. Moto agreed that I take the name of h3spec. Here is an example of execution of h3spec:
% h3spec -v h3spec 0.0.4 % h3spec mew.org 443 QUIC servers MUST send TRANSPORT_PARAMETER_ERROR if initial_source_connection_id is missing [Transport 7.3] MUST send TRANSPORT_PARAMETER_ERROR if original_destination_connection_id is received [Transport 18.2] MUST send TRANSPORT_PARAMETER_ERROR if preferred_address, is received [Transport 18.2] MUST send TRANSPORT_PARAMETER_ERROR if retry_source_connection_id is received [Transport 18.2] MUST send TRANSPORT_PARAMETER_ERROR if stateless_reset_token is received [Transport 18.2] MUST send TRANSPORT_PARAMETER_ERROR if max_udp_payload_size is invalid [Transport 7.4 and 18.2] MUST send FRAME_ENCODING_ERROR if a frame of unknown type is received [Transport 12.4] MUST send PROTOCOL_VIOLATION on no frames [Transport 12.4] MUST send PROTOCOL_VIOLATION if reserved bits in Handshake are non-zero [Transport 17.2] MUST send PROTOCOL_VIOLATION if reserved bits in Short are non-zero [Transport 17.2] MUST send PROTOCOL_VIOLATION if NEW_TOKEN is received [Transport 19.7] MUST send STREAM_STATE_ERROR if MAX_STREAM_DATA is received for a non-existing stream [Transport 19.9] FAILED [1] MUST send PROTOCOL_VIOLATION if HANDSHAKE_DONE is received [Transport 19.20] MUST send no_application_protocol TLS alert if no application protocols are supported [TLS 8.1] MUST the send missing_extension TLS alert if the quic_transport_parameters extension does not included [TLS 8.2] Failures: Transport.hs:60:13: 1) QUIC servers MUST send STREAM_STATE_ERROR if MAX_STREAM_DATA is received for a non-existing stream [Transport 19.9] did not get expected exception: QUICError To rerun use: --match "/QUIC servers/MUST send STREAM_STATE_ERROR if MAX_STREAM_DATA is received for a non-existing stream [Transport 19.9]/" Randomized with seed 1914918977 Finished in 0.7035 seconds 15 examples, 1 failure
I'm planning to add some more error cases. Enjoy!
