Testing Streaming Chatbots with Selenium: Completion, Meaning, and Latency

Selenium can test dynamic interfaces, but the wait condition has to match the application. The presence of a response element or the first visible word does not mean a chatbot has finished answering. Streaming makes that distinction especially easy to miss.

Prefer an explicit completion signal

If the application exposes a completed state, response identifier, or disabled streaming indicator, wait for it and assert the intended response. A DOM quiet period is a useful fallback when that signal is unavailable, but a pause between chunks can look like completion.

The current selenium-chatbot-test README documents StreamWaiter().wait_for_stream_end(driver, locator, silence_timeout=0.5, timeout=30.0). Timeouts are in seconds and the result is a WebElement. The earlier constructor and wait_for_stable_text examples have been replaced here.

from selenium.webdriver.common.by import By
from selenium_chatbot_test import StreamWaiter

# driver is an active session with the target response element present.
response_element = StreamWaiter().wait_for_stream_end(
    driver, (By.ID, "response"), silence_timeout=0.5, timeout=30.0
)
response_text = response_element.text

Treat semantic similarity as one signal

from selenium_chatbot_test import SemanticAssert

SemanticAssert().assert_similarity(
    actual="Hi, how can I assist?",
    expected="Hello, how can I help?",
    min_score=0.7,
)

This is an illustrative threshold, not a promised passing result or 70% semantic accuracy. Similarity can miss a changed number, negation, unsafe instruction, or unsupported factual claim. Calibrate with labeled examples and keep exact checks for required structured fields and prohibited outcomes. The embedding dependency can download a model on first use, so prepare and cache it deliberately in CI.

Measure the visible response lifecycle

LatencyMonitor observes the page around the trigger action. Its first-change timing is a browser-visible proxy for response latency, not a direct measure of the model server’s first token. DOM mutation counts are not tokenizer token counts.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium_chatbot_test import StreamWaiter, LatencyMonitor

def test_chatbot_response():
    driver = webdriver.Chrome()
    try:
        # Replace with an authorized test app whose response node exists
        # before sending and is cleared for each new response.
        driver.get("https://example.com/chat")
        driver.find_element(By.ID, "chat-input").send_keys("Hello")
        with LatencyMonitor(driver, (By.ID, "response")) as monitor:
            driver.find_element(By.ID, "send-btn").click()
            element = StreamWaiter().wait_for_stream_end(
                driver, (By.ID, "response"),
                silence_timeout=0.5, timeout=30.0,
            )
        assert element.text.strip(), "Expected a visible response"
        # Add the application's completion and content assertions here.
        print({"first_change_ms": monitor.metrics.ttft_ms,
               "observed_total_ms": monitor.metrics.total_ms})
    finally:
        driver.quit()

Build the failure set before tuning

• A response that starts late or pauses between chunks.
• A new response that replaces an old DOM node.
• A failed or cancelled request.
• A semantically similar answer containing the wrong value.
• A retry that accidentally appends to a previous response.

Retain response identifiers and sanitized browser evidence so timing failures can be distinguished from content failures. These helpers make checks easier to express; they do not make model output deterministic.

Selenium Chatbot Test current API guide, reviewed September 5, 2026.