Number of prompts: 100
Random_input_length: 128
Random_output_length: 128
| Benchmark Duration | Total input tokens | Total generated tokens | Total generated tokens (retokenized) | Request throughput(req/s) | Input token throughput(tok/s) | Output token throughput(tok/s) | Peak out token throughput | Peak concurrent requests | Total token throughput | Concurrency | Mean E2E Latency (ms) | Median E2E Latency (ms) | P90 E2E Latency(ms) | P99 E2E Latency(ms) | Mean TTFT(ms) | Median TTFT(ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT(ms) | Mean ITL (ms) | Median ITL (ms) | P95 ITL (ms) | P99 ITL (ms) | Max ITL (ms) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 33.99 | 33839 | 21640 | 21631 | 2.94 | 995.69 | 636.74 | 2445 | 100 | 1632.43 | 20.55 | 6985 | 5678.93 | 14130.57 | 19243.26 | 939 | 691.57 | 2321.36 | 65.55 | 31.65 | 441.21 | 28.07 | 23.03 | 33.94 | 79.23 | 1642.27 |
Number of prompts: 200
Random_input_length: 128
Random_output_length: 128
| Benchmark Duration | Total input tokens | Total generated tokens | Total generated tokens (retokenized) | Request throughput(req/s) | Input token throughput(tok/s) | Output token throughput(tok/s) | Peak out token throughput | Peak concurrent requests | Total token throughput | Concurrency | Mean E2E Latency (ms) | Median E2E Latency (ms) | P90 E2E Latency(ms) | P99 E2E Latency(ms) | Mean TTFT(ms) | Median TTFT(ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT(ms) | Mean ITL (ms) | Median ITL (ms) | P95 ITL (ms) | P99 ITL (ms) | Max ITL (ms) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 46.34 | 64205 | 42957 | 42942 | 4.32 | 1385.53 | 927 | 2494 | 200 | 2312.53 | 56.43 | 13074.51 | 12455.86 | 23859 | 31888.06 | 5628.56 | 2751.25 | 15643.06 | 58.23 | 37.42 | 323.20 | 34.83 | 26.41 | 77.89163.52 | 163.52 | 1825.08 |
Number of prompts: 200
Random_input_length: 32
Random_output_length: 128
| Benchmark Duration | Total input tokens | Total generated tokens | Total generated tokens (retokenized) | Request throughput(req/s) | Input token throughput(tok/s) | Output token throughput(tok/s) | Peak out token throughput | Peak concurrent requests | Total token throughput | Concurrency | Mean E2E Latency (ms) | Median E2E Latency (ms) | P90 E2E Latency(ms) | P99 E2E Latency(ms) | Mean TTFT(ms) | Median TTFT(ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT(ms) | Mean ITL (ms) | Median ITL (ms) | P95 ITL (ms) | P99 ITL (ms) | Max ITL (ms) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 47.22 | 64205 | 42957 | 42942 | 4.24 | 1359.64 | 909.68 | 2408 | 200 | 2269.33 | 58.39 | 13786.21 | 12883.17 | 24528.02 | 32772.52 | 6353.37 | 4126.59 | 16001.69 | 66.65 | 35.38 | 477.93 | 34.77 | 26.21 | 73.13 | 150.52 | 3135.28 |
Number of prompts: 200
Random_input_length: 512
Random_output_length: 128
| Benchmark Duration | Total input tokens | Total generated tokens | Total generated tokens (retokenized) | Request throughput(req/s) | Input token throughput(tok/s) | Output token throughput(tok/s) | Peak out token throughput | Peak concurrent requests | Total token throughput | Concurrency | Mean E2E Latency (ms) | Median E2E Latency (ms) | P90 E2E Latency(ms) | P99 E2E Latency(ms) | Mean TTFT(ms) | Median TTFT(ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT(ms) | Mean ITL (ms) | Median ITL (ms) | P95 ITL (ms) | P99 ITL (ms) | Max ITL (ms) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 47.17 | 64205 | 42957 | 42943 | 4.24 | 1361.15 | 910.69 | 2602 | 200 | 2271.84 | 59.33 | 13992.47 | 13155.31 | 24680.14 | 32771.32 | 6470.36 | 4145.97 | 16425.56 | 67.99 | 36.55 | 481.29 | 35.19 | 25.88 | 76.45 | 167.29 | 3189.05 |
Number of prompts: 200
Random_input_length: 1024