Skip to content

Output Tests

Note

Unless otherwise specified, all the continuous batching tests are running with max_model_len=256

Verification of vLLM output by comparing with HF

Run python -m pytest tests/e2e/test_spyre_basic.py.

test_output 

test_output(model: ModelInfo, tp_size: int, backend: str, cb: int, max_num_seqs: int, max_model_len: int, warmup_shapes: DecodeWarmupShapes, monkeypatch: MonkeyPatch, use_llm_cache) -> None

The warmup is based on a single shape. After the warmup, one request with the provided prompts is input to vLLM. The same prompts are also input to HF. The generated output including text, token ids, and logprobs, is verified to be identical for vLLM and HF.

Configuration for CB - parameters are combinatorial
  • max_num_seqs: 4
  • tensor parallelism: 1, 2, 4, 8
  • number of prompts: 4 (Chicken soup prompts)
  • max tokens: 20 (same for all the prompts)
Source code in tests/e2e/test_spyre_basic.py 
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
@pytest.mark.full_model
def test_output(model: ModelInfo, tp_size: int, backend: str, cb: int,
                max_num_seqs: int, max_model_len: int,
                warmup_shapes: DecodeWarmupShapes,
                monkeypatch: pytest.MonkeyPatch, use_llm_cache) -> None:
    '''
    The warmup is based on a single shape. After the warmup,
    one request with the provided prompts is input to vLLM.
    The same prompts are also input to HF. The generated output
    including text, token ids, and logprobs, is verified to be
    identical for vLLM and HF.

    Configuration for CB - parameters are combinatorial:
        * max_num_seqs: 4
        * tensor parallelism: 1, 2, 4, 8
        * number of prompts: 4 (Chicken soup prompts)
        * max tokens: 20 (same for all the prompts)
    '''

    skip_unsupported_tp_size(tp_size, backend)

    prompts = get_chicken_soup_prompts(4)

    kwargs = ({
        "max_num_seqs": max_num_seqs,
        "use_cb": True,
    } if cb == 1 else {
        "warmup_shapes": warmup_shapes,
    })

    max_new_tokens = warmup_shapes[0][1]

    vllm_sampling_params = SamplingParams(
        max_tokens=max_new_tokens,
        temperature=0,
        logprobs=0,  # return logprobs of generated tokens only
        ignore_eos=True)

    vllm_results = generate_spyre_vllm_output(
        model=model,
        prompts=prompts,
        sampling_params=vllm_sampling_params,
        tensor_parallel_size=tp_size,
        backend=backend,
        monkeypatch=monkeypatch,
        max_model_len=max_model_len,
        **kwargs)
    check_output_against_hf(model, backend, max_new_tokens, vllm_results,
                            prompts)

test_batch_handling 

test_batch_handling(model: ModelInfo, backend: str, cb: int, warmup_shapes, max_num_seqs: int, max_model_len: int, monkeypatch: MonkeyPatch, use_llm_cache)

Test that the spyre worker correctly handles continuous batches of requests that finish after different numbers of forward passes

Configuration for CB - parameters are combinatorial
  • max_num_seqs: 2
  • number of prompts: 4 (Chicken soup prompts)
  • max tokens: [5, 20, 10, 5]
Source code in tests/e2e/test_spyre_basic.py 
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
def test_batch_handling(model: ModelInfo, backend: str, cb: int, warmup_shapes,
                        max_num_seqs: int, max_model_len: int,
                        monkeypatch: pytest.MonkeyPatch, use_llm_cache):
    """Test that the spyre worker correctly handles
    continuous batches of requests that
    finish after different numbers of forward passes

    Configuration for CB - parameters are combinatorial:
        * max_num_seqs: 2
        * number of prompts: 4 (Chicken soup prompts)
        * max tokens: [5, 20, 10, 5]
    """

    prompts = get_chicken_soup_prompts(4)

    max_new_tokens = [5, 20, 10, 5]

    vllm_sampling_params = [
        SamplingParams(max_tokens=max_new_tokens[i],
                       min_tokens=max_new_tokens[i],
                       temperature=0,
                       ignore_eos=True,
                       logprobs=0) for i in range(len(max_new_tokens))
    ]

    kwargs = {
        "max_num_seqs": max_num_seqs,
        "use_cb": True
    } if cb == 1 else {
        "warmup_shapes": warmup_shapes
    }

    vllm_results = generate_spyre_vllm_output(
        model=model,
        prompts=prompts,
        max_model_len=256,
        sampling_params=vllm_sampling_params,
        tensor_parallel_size=1,
        backend=backend,
        monkeypatch=monkeypatch,
        **kwargs)

    check_output_against_hf(model, backend, max_new_tokens, vllm_results,
                            prompts)