the solar cell in terms of $A, R_{\\\\odot}, d_{\\\\odot}, T_{\\\\mathrm{s}}, v$ and the fundamental constants $c, h, k_{\\\\mathrm{B}}$.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: ee38c137525dddde\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/olympiads\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"To determine the parameters \\\\( a_{1}, a_{2}, \\\\) and \\\\( a_{3} \\\\) in the formula \\\\( y=a_{1} x^{2}+a_{2} x+a_{3} \\\\), values of \\\\( y \\\\) were measured at various values of \\\\( x \\\\). The sample obtained is:\\n\",\n \"\\n\",\n \"\\\\[\\n\",\n \"\\\\begin{array}{rrrrrrrrrr}\\n\",\n \"x_{k} & -1 & -0.75 & -0.5 & -0.25 & 0 & 0.25 & 0.5 & 0.75 & 1 \\\\\\\\\\n\",\n \"y_{k} & 6.01 & 5.07 & 4.30 & 3.56 & 3.07 & 2.87 & 2.18 & 2.00 & 2.14\\n\",\n \"\\\\end{array}\\n\",\n \"\\\\]\\n\",\n \"\\n\",\n \"Using these data, determine the parameters \\\\( a_{1}, a_{2} \\\\), and \\\\( a_{3} \\\\) by the method of least squares. Measurement errors of \\\\( x_{1}, \\\\ldots, x_{9} \\\\) and computational inaccuracies can be neglected.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 2b8218fdf3bda4e5\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Tetrahedron $ABCD$ has $AD=BC=28$ , $AC=BD=44$ , and $AB=CD=52$ . For any point $X$ in space, define $f(X)=AX+BX+CX+DX$ . The least possible value of $f(X)$ can be expressed as $m\\\\sqrt{n}$ , where $m$ and $n$ are positive integers, and $n$ is not divisible by the square of any prime. Find $m+n$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: f86027628b14dafe\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"The formula for converting a Fahrenheit temperature $F$ to the corresponding Celsius temperature $C$ is $C = \\\\frac{5}{9}(F-32).$ An integer Fahrenheit temperature is converted to Celsius, rounded to the nearest integer, converted back to Fahrenheit, and again rounded to the nearest integer. For how many integer Fahrenheit temperatures between $32$ and $1000$ inclusive does the original temperature equal the final temperature?\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 2213a3433344dec5\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Point $P$ lies on the diagonal $AC$ of square $ABCD$ with $AP>CP$ . Let $O_1$ and $O_2$ be the circumcenters of triangles $ABP$ and $CDP$ respectively. Given that $AB=12$ and $\\\\angle O_1 P O_2 = 120^\\\\circ$ , then $AP=\\\\sqrt{a}+\\\\sqrt{b}$ where $a$ and $b$ are positive integers. Find $a+b$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 6c48f6ffa1ce97a3\\n\",\n \"## source_type: qfq/openaimath/Precalculus\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"In a right triangle, one of the acute angles $\\\\alpha$ satisfies\\n\",\n \"\\\\[\\\\tan \\\\frac{\\\\alpha}{2} = \\\\frac{1}{\\\\sqrt[3]{2}}.\\\\]Let $\\\\theta$ be the angle between the median and the angle bisector drawn from this acute angle. Find $\\\\tan \\\\theta.$\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 6ab3b964c7e4c75a\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"The Fibonacci numbers $F_1, F_2, F_3, \\\\ldots$ are defined by $F_1 = F_2 = 1$ and $F_{n+2} = F_{n+1} + F_n$ for each integer $n \\\\ge 1$ . Let $P$ be the unique polynomial of least degree for which $P(n) = F_n$ for all integers $1 \\\\le n \\\\le 10$ . Compute the integer $m$ for which \\\\[P(100) - \\\\sum_{k=11}^{98} P(k) = \\\\frac{m}{10} \\\\dbinom{98}{9} + 144.\\\\]\\n\",\n \"\\n\",\n \"*Proposed by Michael Tang*\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 7a6816402d6e5607\\n\",\n \"## source_type: qfq/openaimath/Precalculus\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Let $\\\\mathbf{a},$ $\\\\mathbf{b},$ $\\\\mathbf{c},$ $\\\\mathbf{d}$ be four distinct unit vectors in space such that\\n\",\n \"\\\\[\\\\mathbf{a} \\\\cdot \\\\mathbf{b} = \\\\mathbf{a} \\\\cdot \\\\mathbf{c} = \\\\mathbf{b} \\\\cdot \\\\mathbf{c} =\\\\mathbf{b} \\\\cdot \\\\mathbf{d} = \\\\mathbf{c} \\\\cdot \\\\mathbf{d} = -\\\\frac{1}{11}.\\\\]Find $\\\\mathbf{a} \\\\cdot \\\\mathbf{d}.$\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 950d9b30fa76fbb2\\n\",\n \"## source_type: qfq/openaimath/Precalculus\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"In triangle $ABC,$ let $D,$ $E,$ $F$ be the midpoints of $\\\\overline{BC},$ $\\\\overline{AC},$ $\\\\overline{AB},$ respectively. Let $P,$ $Q,$ $R$ be the midpoints of $\\\\overline{AD},$ $\\\\overline{BE},$ $\\\\overline{CF},$ respectively. Compute\\n\",\n \"\\\\[\\\\frac{AQ^2 + AR^ 2+ BP^2 + BR^2 + CP^2 + CQ^2}{AB^2 + AC^2 + BC^2}.\\\\]\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 9f7f1563a23537aa\\n\",\n \"## source_type: baber/agieval/math_agieval\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Let $\\\\lambda$ be a constant, $0 \\\\le \\\\lambda \\\\le 4,$ and let $f : [0,1] \\\\to [0,1]$ be defined by\\n\",\n \"\\\\[f(x) = \\\\lambda x(1 - x).\\\\]Find the values of $\\\\lambda,$ $0 \\\\le \\\\lambda \\\\le 4,$ for which there exists an $x \\\\in [0,1]$ such that $f(x) \\\\neq x$ but $f(f(x)) = x.$\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 37f719cc9e67e64b\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Three clever monkeys divide a pile of bananas. The first monkey takes some bananas from the pile, keeps three-fourths of them, and divides the rest equally between the other two. The second monkey takes some bananas from the pile, keeps one-fourth of them, and divides the rest equally between the other two. The third monkey takes the remaining bananas from the pile, keeps one-twelfth of them, and divides the rest equally between the other two. Given that each monkey receives a whole number of bananas whenever the bananas are divided, and the numbers of bananas the first, second, and third monkeys have at the end of the process are in the ratio $3: 2: 1,$ what is the least possible total for the number of bananas?\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 5cdb4c6d0c15130a\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Let $P$ be a square pyramid whose base consists of the four vertices $(0, 0, 0), (3, 0, 0), (3, 3, 0)$ , and $(0, 3, 0)$ , and whose apex is the point $(1, 1, 3)$ . Let $Q$ be a square pyramid whose base is the same as the base of $P$ , and whose apex is the point $(2, 2, 3)$ . Find the volume of the intersection of the interiors of $P$ and $Q$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 30a96a6f8e988a3e\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Let $S$ be the number of ordered pairs of integers $(a,b)$ with $1 \\\\leq a \\\\leq 100$ and $b \\\\geq 0$ such that the polynomial $x^2+ax+b$ can be factored into the product of two (not necessarily distinct) linear factors with integer coefficients. Find the remainder when $S$ is divided by $1000$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 58d8003c0f2a224b\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Determine all positive integers $n\\\\geq 2$ for which there exist integers $x_1,x_2,\\\\ldots ,x_{n-1}$ satisfying the condition that if $0
value.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 08f3a7a834d4be1b\\n\",\n \"## source_type: Idavidrein/gpqa\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"A compound with the molecular formula C8H6O2 gives a positive DNP test, a positive silver mirror test, and also reacts with KMnO4 to give benzoic acid. Which one of the following proton NMR data corresponds to the given compound?\\n\",\n \"Answer Choices:\\n\",\n \"(A) 1H NMR (ppm): 9.88 (s, 2H), 8.08 (s, 4H).\\n\",\n \"(B) 1H NMR (ppm): 9.88 (s, 2H), 8.17 (d, 2H), 7.72 (t, 1H), and 7.63 (s, 1H).\\n\",\n \"(C) 1H NMR (ppm): 10.36 (s, 2H), 8.08 (d, 2H), 7.92 (d, 2H)\\n\",\n \"(D) 1H NMR (ppm): 9.48 (s, 1H), 7.89 (d, 2H), 7.64 (m, 2H), 7.73 (m, 1H)\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: d7a6393dad8c3a16\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Triangle $ABC$ is isosceles with $AC = BC$ and $\\\\angle ACB = 106^\\\\circ.$ Point $M$ is in the interior of the triangle so that $\\\\angle MAC = 7^\\\\circ$ and $\\\\angle MCA = 23^\\\\circ.$ Find the number of degrees in $\\\\angle CMB.$\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: d9edf8131036e885\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"A container in the shape of a right circular cone is 12 inches tall and its base has a 5-inch radius. The liquid that is sealed inside is 9 inches deep when the cone is held with its point down and its base horizontal. When the liquid is held with its point up and its base horizontal, the height of the liquid is $m - n\\\\sqrt [3]{p},$ where $m,$ $n,$ and $p$ are positive integers and $p$ is not divisible by the cube of any prime number. Find $m + n + p$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 444b959c049d1311\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Given a triangle with acute angle $ \\\\angle BEC,$ let $ E$ be the midpoint of $ AB.$ Point $ M$ is chosen on the opposite ray of $ EC$ such that $ \\\\angle BME \\\\equal{} \\\\angle ECA.$ Denote by $ \\\\theta$ the measure of angle $ \\\\angle BEC.$ Evaluate $ \\\\frac{MC}{AB}$ in terms of $ \\\\theta.$ \\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 760635ba833444fd\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Circle $C$ with radius 2 has diameter $\\\\overline{AB}$ . Circle $D$ is internally tangent to circle $C$ at $A$ . Circle $E$ is internally tangent to circle $C$ , externally tangent to circle $D$ , and tangent to $\\\\overline{AB}$ . The radius of circle $D$ is three times the radius of circle $E$ , and can be written in the form $\\\\sqrt{m}-n$ , where $m$ and $n$ are positive integers. Find $m+n$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 51ac4d1c6147a8fa\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Given integer $m\\\\geq2$ , $x_1,...,x_m$ are non-negative real numbers,prove that: $$ (m-1)^{m-1}(x_1^m+...+x_m^m)\\\\geq(x_1+...+x_m)^m-m^mx_1...x_m $$ and please find out when the equality holds.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: c44d2bf7affbe631\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Given a real number $x,$ let $\\\\lfloor x \\\\rfloor$ denote the greatest integer less than or equal to $x.$ For a certain integer $k,$ there are exactly $70$ positive integers $n_{1}, n_{2}, \\\\ldots, n_{70}$ such that $k=\\\\lfloor\\\\sqrt[3]{n_{1}}\\\\rfloor = \\\\lfloor\\\\sqrt[3]{n_{2}}\\\\rfloor = \\\\cdots = \\\\lfloor\\\\sqrt[3]{n_{70}}\\\\rfloor$ and $k$ divides $n_{i}$ for all $i$ such that $1 \\\\leq i \\\\leq 70.$ Find the maximum value of $\\\\frac{n_{i}}{k}$ for $1\\\\leq i \\\\leq 70.$\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: ffc2777ae9e0cf20\\n\",\n \"## source_type: Idavidrein/gpqa\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"On a hot summer day (30 degrees Celsius), a chemist received a set of samples of three metals that were similar in appearance (color) and found a way to distinguish them. To do this, he exposed the samples to the action of a concentrated solution of acids and sodium hydroxide solution. The results of his research can be presented in the form of the following table, where the “+” sign means cases when the metal dissolves, and the “-” sign means cases when the metal does not react with the acid or alkali solution:\\n\",\n \"Reagent/ Metal №1 Metal №2 Metal №3\\n\",\n \"Hydrochloric Acid/ - + +\\n\",\n \"Nitric Acid/ + - +\\n\",\n \"Sodium Hydroxide solution/ - + +\\n\",\n \"Note. Three separate samples of each metal were tested separately with three different reagents.\\n\",\n \"\\n\",\n \"Add the atomic weights of all three metals and indicate the range in which they fall.\\n\",\n \"Answer Choices:\\n\",\n \"(A) from 130 to 150\\n\",\n \"(B) from 220 to 240\\n\",\n \"(C) from 190 to 210\\n\",\n \"(D) from 160 to 180\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 914dd9e8da28a36d\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Find the eigenvalues and their multiplicities of the Laplace operator $\\\\Delta = \\\\text{div grad}$ on a sphere of radius $R$ in Euclidean space of dimension $n$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 0b6d429d407a2a60\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Circles $\\\\mathcal{C}_{1}$ and $\\\\mathcal{C}_{2}$ intersect at two points, one of which is $(9,6)$ , and the product of the radii is $68$ . The x-axis and the line $y = mx$ , where $m > 0$ , are tangent to both circles. It is given that $m$ can be written in the form $a\\\\sqrt {b}/c$ , where $a$ , $b$ , and $c$ are positive integers, $b$ is not divisible by the square of any prime, and $a$ and $c$ are relatively prime. Find $a + b + c$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 903e6c9abc4fad43\\n\",\n \"## source_type: Hothan/OlympiadBench/Open-ended/Physics\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"In the far future, the Earth received an enormous amount of charge as a result of Mad Scientist ecilA's nefarious experiments. Specifically, the total charge on Earth is $Q=1.0 \\\\times 10^{11} \\\\mathrm{C}$. (compare this with the current $5 \\\\times 10^{5} \\\\mathrm{C}$ ).\\n\",\n \"Estimate the maximum height of a \\\"mountain\\\" on Earth that has a circular base with diameter $w=1.0$ $\\\\mathrm{km}$, if it has the shape of a spherical sector. You may assume that $h_{\\\\max } \\\\ll w$. The tensile strength of rock is $10 \\\\mathrm{MPa}$.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: aafe1efafccd0b50\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"A straight river that is $264$ meters wide flows from west to east at a rate of $14$ meters per minute. Melanie and Sherry sit on the south bank of the river with Melanie a distance of $D$ meters downstream from Sherry. Relative to the water, Melanie swims at $80$ meters per minute, and Sherry swims at $60$ meters per minute. At the same time, Melanie and Sherry begin swimming in straight lines to a point on the north bank of the river that is equidistant from their starting positions. The two women arrive at this point simultaneously. Find $D.$\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: e7dc378a65fe121c\\n\",\n \"## source_type: Hothan/OlympiadBench/Open-ended/Physics\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"(b) Any real inductor has undesired, or parasitic, resistance. We can model the real inductor as an ideal inductor $L$ in series with a parasitic resistance $R$.\\n\",\n \"\\n\",\n \"Due to the thermal noise $\\\\frac{d\\\\left\\\\langle V^{2}\\\\right\\\\rangle}{d f}=4 k T R$ of its parasitic resistance, this (real) inductor will support a nonzero per-frequency mean-squared current, $\\\\frac{d\\\\left\\\\langle I^{2}\\\\right\\\\rangle}{d f}$, even when both sides of the inductor are grounded. Compute $\\\\frac{d\\\\left\\\\langle I^{2}\\\\right\\\\rangle}{d f}$ as a function of $f, L, T$ and $R$.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 349047e567587b90\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Let $P(x)$ be a nonzero polynomial such that $(x-1)P(x+1)=(x+2)P(x)$ for every real $x$ , and $\\\\left(P(2)\\\\right)^2 = P(3)$ . Then $P(\\\\tfrac72)=\\\\tfrac{m}{n}$ , where $m$ and $n$ are relatively prime positive integers. Find $m + n$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 7cc28a494d0a4538\\n\",\n \"## source_type: qfq/openaimath/Counting & Probability\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"For $\\\\{1, 2, 3, \\\\ldots, n\\\\}$ and each of its non-empty subsets a unique alternating sum is defined as follows. Arrange the numbers in the subset in decreasing order and then, beginning with the largest, alternately add and subtract successive numbers. For example, the alternating sum for $\\\\{1, 2, 3, 6,9\\\\}$ is $9-6+3-2+1=5$ and for $\\\\{5\\\\}$ it is simply $5$. Find the sum of all such alternating sums for $n=7$.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: a1148db9187b8b62\\n\",\n \"## source_type: daman1209arora/jeebench/chem\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"The boiling point of water in a 0.1 molal silver nitrate solution (solution $\\\\mathbf{A}$ ) is $\\\\mathbf{x}^{\\\\circ} \\\\mathrm{C}$. To this solution $\\\\mathbf{A}$, an equal volume of 0.1 molal aqueous barium chloride solution is added to make a new solution $\\\\mathbf{B}$. The difference in the boiling points of water in the two solutions $\\\\mathbf{A}$ and $\\\\mathbf{B}$ is $\\\\mathbf{y} \\\\times 10^{-2}{ }^{\\\\circ} \\\\mathrm{C}$.\\n\",\n \"(Assume: Densities of the solutions $\\\\mathbf{A}$ and $\\\\mathbf{B}$ are the same as that of water and the soluble salts dissociate completely.\\n\",\n \"Use: Molal elevation constant (Ebullioscopic Constant), $K_b=0.5 \\\\mathrm{~K} \\\\mathrm{~kg} \\\\mathrm{~mol}^{-1}$; Boiling point of pure water as $100^{\\\\circ} \\\\mathrm{C}$.) What is the value of $|\\\\mathbf{y}|$?\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: d73758dd2431d32f\\n\",\n \"## source_type: KbsdJames/Omni-MATH\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Svitlana writes the number 147 on a blackboard. Then, at any point, if the number on the blackboard is $n$, she can perform one of the following three operations: - if $n$ is even, she can replace $n$ with $\\\\frac{n}{2}$; - if $n$ is odd, she can replace $n$ with $\\\\frac{n+255}{2}$; and - if $n \\\\geq 64$, she can replace $n$ with $n-64$. Compute the number of possible values that Svitlana can obtain by doing zero or more operations.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 3af4dd9ea22a4efa\\n\",\n \"## source_type: Idavidrein/gpqa\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"You would like to use PCR to amplify the following sequence (written from 5’ to 3’ direction) for subcloning into an expression vector. In order to clone it in the correct orientation for subsequent expression, you decide to clone it using two different restriction enzymes.\\n\",\n \"\\n\",\n \"ATGAATCCGCGGCAGGGGTATTCCCTCAGCGGATACTACACCCATCCATTTCAAG\\n\",\n \"GCTATGAGCACAGACAGCTCAGGTACCAGCAGCCTGGGCCAGGATCTTCCCCCAG\\n\",\n \"TAGTTTCCTGCTTAAGCAAATAGAATTTCTCAAGGGGCAGCTCCCAGAAGCACCGG\\n\",\n \"TGATTGGAAAGCAGACACCGTCACTGCCACCTTCCCTCCCAGGACTCCGGCCAAG\\n\",\n \"GTTTCCAGTACTACTTGCCTCCAGTACCAGAGGCAGGCAAGTGGACATCAGGGGT\\n\",\n \"GTCCCCAGGGGCGTGCATCTCGGAAGTCAGGGGCTCCAGAGAGGGTTCCAGCAT\\n\",\n \"CCTTCACCACGTGGCAGGAGTCTGCCACAGAGAGGTGTTGATTGCCTTTCCTCAC\\n\",\n \"ATTTCCAGGAACTGAGTATCTACCAAGATCAGGAACAAAGGATCTTAAAGTTCCTG\\n\",\n \"GAAGAGCTTGGGGAAGGGAAGGCCACCACAGCACATGATCTGTCTGGGAAACTTG\\n\",\n \"GGACTCCGAAGAAAGAAATCAATCGAGTTTTATACTCCCTGGCAAAGAAGGGCAAG\\n\",\n \"CTACAGAAAGAGGCAGGAACACCCCCTTTGTGGAAAATCGCGGTCTC\\n\",\n \"\\n\",\n \"You have the enzymes ApaI, BamHI, BstNI, HindIII, KpnI, and ScaI in your freezer.\\n\",\n \"The order of features on your expression vector looks like this (the cleavage sites in MCS are shown within brackets):\\n\",\n \"\\n\",\n \"--CMV promoter---[SmaI--HindIII--BglII--KpnI---PstI--BamHI--EcoRI]-------\\n\",\n \"\\n\",\n \"Using the information you have been given, choose a suitable pair of primers for PCR (shown in the 5’ to 3’ direction). Ensure your primers will make it possible to clone the sequence given above into the expression vector in a directional manner.\\n\",\n \"Answer Choices:\\n\",\n \"(A) aaaaggatccATGAATCCGCGGCAGGG; aaaaagcttGAGACCGCGATTTTCCACAAAGG\\n\",\n \"(B) aaaaaagcttATGAATCCGCGGCAGGG; aaaggtaccGAGACCGCGATTTTCCACAAAGG\\n\",\n \"(C) aaaaaagcttATGAATCCGCGGCAGGG; aaaggatccGAGACCGCGATTTTCCACAAAGG\\n\",\n \"(D) aaaacccgggATGAATCCGCGGCAGGG; aaaggatccGAGACCGCGATTTTCCACAAAGG\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 84454560a79a90e4\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Let $ ABC$ be an equilateral triangle of center $ O$ , and $ M\\\\in BC$ . Let $ K,L$ be projections of $ M$ onto the sides $ AB$ and $ AC$ respectively. Prove that line $ OM$ passes through the midpoint of the segment $ KL$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 4396b7ccc09ddf68\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"(a)\\tDecide whether there exist two decimal digits $a$ and $b$ , such that every integer with decimal representation $ab222 ... 231$ is divisible by $73$ .\\n\",\n \"(b)\\tDecide whether there exist two decimal digits $c$ and $d$ , such that every integer with decimal representation $cd222... 231$ is divisible by $79$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 13eb9d92b2babbf6\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"The points $A$ , $B$ and $C$ lie on the surface of a sphere with center $O$ and radius $20$ . It is given that $AB=13$ , $BC=14$ , $CA=15$ , and that the distance from $O$ to triangle $ABC$ is $\\\\frac{m\\\\sqrt{n}}k$ , where $m$ , $n$ , and $k$ are positive integers, $m$ and $k$ are relatively prime, and $n$ is not divisible by the square of any prime. Find $m+n+k$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: e6301ec8f6c7d312\\n\",\n \"## source_type: Idavidrein/gpqa\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"A collimated beam emerges from an analyzing spectrometer with E=4GeV. The beam is composed of particle X, with m{X}=1.48GeV and \\\\tau{0}=8.7\\\\times10^{-5}s. At what distance is the flux reduced to 25%?\\n\",\n \"\\n\",\n \"PS: Use a LaTeX online editor for the maths.\\n\",\n \"Answer Choices:\\n\",\n \"(A) 90954 m\\n\",\n \"(B) 48635 m\\n\",\n \"(C) 64218 m\\n\",\n \"(D) 17586 m\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 5d5fbc08bbed4af6\\n\",\n \"## source_type: Hothan/OlympiadBench/Open-ended/Physics\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"1.2.1. Write down the expression giving the atmospheric pressure $p$ as a function of the altitude $Z$.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 0f7b165fa4540e6e\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Alpha and Beta both took part in a two-day problem-solving competition. At the end of the second day, each had attempted questions worth a total of $500$ points. Alpha scored $160$ points out of $300$ points attempted on the first day, and scored $140$ points out of $200$ points attempted on the second day. Beta who did not attempt $300$ points on the first day, had a positive integer score on each of the two days, and Beta's daily success rate (points scored divided by points attempted) on each day was less than Alpha's on that day. Alpha's two-day success ratio was $\\\\frac{300}{500} = \\\\frac{3}{5}$ . The largest possible two-day success ratio that Beta could achieve is $m/n,$ where $m$ and $n$ are relatively prime positive integers. What is $m+n$ ?\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 2958d2d518405ee2\\n\",\n \"## source_type: qfq/openaimath/Precalculus\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"In triangle $ABC$, $AB = 13$, $BC = 15$, and $CA = 14$. Point $D$ is on $\\\\overline{BC}$ with $CD = 6$. Point $E$ is on $\\\\overline{BC}$ such that $\\\\angle BAE = \\\\angle CAD$. Find $BE.$\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 434924513f213b98\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"A hexagon that is inscribed in a circle has side lengths $22$ , $22$ , $20$ , $22$ , $22$ , and $20$ in that order. The radius of the circle can be written as $p+\\\\sqrt{q}$ , where $p$ and $q$ are positive integers. Find $p+q$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 3dff6dca15751dea\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"A set $\\\\mathcal{S}$ of distinct positive integers has the following property: for every integer $x$ in $\\\\mathcal{S},$ the arithmetic mean of the set of values obtained by deleting $x$ from $\\\\mathcal{S}$ is an integer. Given that 1 belongs to $\\\\mathcal{S}$ and that 2002 is the largest element of $\\\\mathcal{S},$ what is the greatest number of elements that $\\\\mathcal{S}$ can have?\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: dc5e70c8f6617cf8\\n\",\n \"## source_type: Idavidrein/gpqa\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Enya and John are of normal phenotype but they have a son with a genetic condition that is caused by a mutation in the DMD gene and is a sex-linked recessive disorder. What are the chances of having a carrier and diseased child with each pregnancy if the sex of the child is unknown and there were no epigenetic changes involved in the progression of the disease?\\n\",\n \"Answer Choices:\\n\",\n \"(A) 50% carrier and 50% diseased\\n\",\n \"(B) 100% carrier and 50% diseased\\n\",\n \"(C) 25% carrier and 25% diseased\\n\",\n \"(D) 50% carrier and 25% diseased\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: ec5d9654107ab668\\n\",\n \"## source_type: KbsdJames/Omni-MATH\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"For a continuous and absolutely integrable complex-valued function $f(x)$ on $\\\\mathbb{R}$, define a function $(S f)(x)$ on $\\\\mathbb{R}$ by $(S f)(x)=\\\\int_{-\\\\infty}^{+\\\\infty} e^{2 \\\\pi \\\\mathrm{i} u x} f(u) \\\\mathrm{d} u$. Find explicit forms of $S\\\\left(\\\\frac{1}{1+x^{2}}\\\\right)$ and $S\\\\left(\\\\frac{1}{\\\\left(1+x^{2}\\\\right)^{2}}\\\\right)$.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: f2025557f2cbd011\\n\",\n \"## source_type: qfq/openaimath/Intermediate Algebra\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Let a sequence be defined as follows: $a_1 = 3,$ $a_2 = 3,$ and for $n \\\\ge 2,$\\n\",\n \"\\\\[a_{n + 1} a_{n - 1} = a_n^2 + 2007.\\\\]Find the largest integer less than or equal to $\\\\frac{a_{2007}^2+a_{2006}^2}{a_{2007}a_{2006}}$.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 74170d1655e58eed\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Let $\\\\tau(n)$ denote the number of positive integer divisors of $n$ . Find the sum of the six least positive integers $n$ that are solutions to $\\\\tau (n) + \\\\tau (n+1) = 7$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 21eed436c6e75084\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Suppose that $a$ , $b$ , and $c$ are positive real numbers such that $a^{\\\\log_3 7} = 27$ , $b^{\\\\log_7 11} = 49$ , and $c^{\\\\log_{11}25} = \\\\sqrt{11}$ . Find \\\\[a^{(\\\\log_3 7)^2} + b^{(\\\\log_7 11)^2} + c^{(\\\\log_{11} 25)^2}.\\\\]\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 9025d78ae2c60023\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"In triangle $ABC$ , $AB = 125$ , $AC = 117$ , and $BC = 120$ . The angle bisector of angle $A$ intersects $\\\\overline{BC}$ at point $L$ , and the angle bisector of angle $B$ intersects $\\\\overline{AC}$ at point $K$ . Let $M$ and $N$ be the feet of the perpendiculars from $C$ to $\\\\overline{BK}$ and $\\\\overline{AL}$ , respectively. Find $MN$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: fd6d3f9a996b040f\\n\",\n \"## source_type: Idavidrein/gpqa\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Arrange given compounds (1. Acetophenone, 2. propane-2,2-diyldibenzene, 3. Styrene, 4. 1-oxaspiro[4.4]nonane) in increasing oxidation state of central carbon atom (A). Also, select the proper sequence of reaction when an ester is converted first into an alcohol and then into an acid (B).\\n\",\n \"\\n\",\n \"1. Oxidizing reagent followed by reducing reagent\\n\",\n \"2. Reducing reagent followed by oxidizing reagent\\n\",\n \"Answer Choices:\\n\",\n \"(A) A = 2, 3, 1, and 4, B = 2\\n\",\n \"(B) A = 3, 2, 4, and 1, B = 2\\n\",\n \"(C) A = 2, 3, 1, and 4, B = 1\\n\",\n \"(D) A = 3, 2, 4, and 1, B = 1\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: a2259bec7a70b7e4\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"The real root of the equation $8x^3-3x^2-3x-1=0$ can be written in the form $\\\\frac{\\\\sqrt[3]{a}+\\\\sqrt[3]{b}+1}{c}$ , where $a$ , $b$ , and $c$ are positive integers. Find $a+b+c$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: c4e68a57fbacd618\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Call a three-term strictly increasing arithmetic sequence of integers special if the sum of the squares of the three terms equals the product of the middle term and the square of the common difference. Find the sum of the third terms of all special sequences. \\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 4832c8a27909b8c7\\n\",\n \"## source_type: KbsdJames/Omni-MATH\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Von Neumann's Poker: The first step in Von Neumann's game is selecting a random number on $[0,1]$. To generate this number, Chebby uses the factorial base: the number $0 . A_{1} A_{2} A_{3} A_{4} \\\\ldots$ stands for $\\\\sum_{n=0}^{\\\\infty} \\\\frac{A_{n}}{(n+1)!}$, where each $A_{n}$ is an integer between 0 and $n$, inclusive. Chebby has an infinite number of cards labeled $0, 1, 2, \\\\ldots$. He begins by putting cards $0$ and $1$ into a hat and drawing randomly to determine $A_{1}$. The card assigned $A_{1}$ does not get reused. Chebby then adds in card 2 and draws for $A_{2}$, and continues in this manner to determine the random number. At each step, he only draws one card from two in the hat. Unfortunately, this method does not result in a uniform distribution. What is the expected value of Chebby's final number?\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: f7010ac204354ee8\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Given a positive integer $k$ , find the least integer $n_k$ for which there exist five sets $S_1, S_2, S_3, S_4, S_5$ with the following properties:\\n\",\n \"\\\\[|S_j|=k \\\\text{ for } j=1, \\\\cdots , 5 , \\\\quad |\\\\bigcup_{j=1}^{5} S_j | = n_k ;\\\\]\\n\",\n \"\\\\[|S_i \\\\cap S_{i+1}| = 0 = |S_5 \\\\cap S_1|, \\\\quad \\\\text{for } i=1,\\\\cdots ,4 \\\\]\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: c2fc8504c94d6a75\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Find the number of permutations $x_1, x_2, x_3, x_4, x_5$ of numbers $1, 2, 3, 4, 5$ such that the sum of five products \\\\[x_1x_2x_3 + x_2x_3x_4 + x_3x_4x_5 + x_4x_5x_1 + x_5x_1x_2\\\\] is divisible by $3$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 5f5c112a268e5ed3\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"For the integer $n>1$ , define $D(n)=\\\\{ a-b\\\\mid ab=n, a>b>0, a,b\\\\in\\\\mathbb{N} \\\\}$ . Prove that for any integer $k>1$ , there exists pairwise distinct positive integers $n_1,n_2,\\\\ldots,n_k$ such that $n_1,\\\\ldots,n_k>1$ and $|D(n_1)\\\\cap D(n_2)\\\\cap\\\\cdots\\\\cap D(n_k)|\\\\geq 2$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 4dc52341c1bd27f4\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Let $w$ and $z$ be complex numbers such that $|w| = 1$ and $|z| = 10$ . Let $\\\\theta = \\\\arg \\\\left(\\\\tfrac{w-z}{z}\\\\right)$ . The maximum possible value of $\\\\tan^2 \\\\theta$ can be written as $\\\\tfrac{p}{q}$ , where $p$ and $q$ are relatively prime positive integers. Find $p+q$ . (Note that $\\\\arg(w)$ , for $w \\\\neq 0$ , denotes the measure of the angle that the ray from $0$ to $w$ makes with the positive real axis in the complex plane.)\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 59696d411cd25bd4\\n\",\n \"## source_type: AI-MO/NuminaMath-CoT/aops_forum\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Let $ABCDEF$ be a regular hexagon and $P$ be a point on the shorter arc $EF$ of its circumcircle. Prove that the value of $$ \\\\frac{AP+BP+CP+DP}{EP+FP} $$ is constant and find its value.\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 9fdd953f846e56d0\\n\",\n \"## source_type: qfq/openaimath/Precalculus\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"We define two sequences of vectors $(\\\\mathbf{v}_n)$ and $(\\\\mathbf{w}_n)$ as follows: First, $\\\\mathbf{v}_0 = \\\\begin{pmatrix} 1 \\\\\\\\ 3 \\\\end{pmatrix},$ $\\\\mathbf{w}_0 = \\\\begin{pmatrix} 4 \\\\\\\\ 0 \\\\end{pmatrix}.$ Then for all $n \\\\ge 1,$ $\\\\mathbf{v}_n$ is the projection of $\\\\mathbf{w}_{n - 1}$ onto $\\\\mathbf{v}_0,$ and $\\\\mathbf{w}_n$ is the projection of $\\\\mathbf{v}_n$ onto $\\\\mathbf{w}_0.$ Find\\n\",\n \"\\\\[\\\\mathbf{v}_1 + \\\\mathbf{v}_2 + \\\\mathbf{v}_3 + \\\\dotsb.\\\\]\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: f8f43679754aa59e\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"For a permutation $p = (a_1,a_2,\\\\ldots,a_9)$ of the digits $1,2,\\\\ldots,9$ , let $s(p)$ denote the sum of the three $3$ -digit numbers $a_1a_2a_3$ , $a_4a_5a_6$ , and $a_7a_8a_9$ . Let $m$ be the minimum value of $s(p)$ subject to the condition that the units digit of $s(p)$ is $0$ . Let $n$ denote the number of permutations $p$ with $s(p) = m$ . Find $|m - n|$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 78ec502828fb712d\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Let $p(x,y) = a_0 + a_1x + a_2y + a_3x^2 + a_4xy + a_5y^2 + a_6x^3 + a_7x^2y + a_8xy^2 + a_9y^3$ . Suppose that $p(0,0) = p(1,0) = p( - 1,0) = p(0,1) = p(0, - 1) = p(1,1) = p(1, - 1) = p(2,2) = 0$ . There is a point $\\\\left(\\\\frac {a}{c},\\\\frac {b}{c}\\\\right)$ for which $p\\\\left(\\\\frac {a}{c},\\\\frac {b}{c}\\\\right) = 0$ for all such polynomials, where $a$ , $b$ , and $c$ are positive integers, $a$ and $c$ are relatively prime, and $c > 1$ . Find $a + b + c$ .\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 2ef1d89f465f8145\\n\",\n \"## source_type: Idavidrein/gpqa\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Astronomers are studying a star with a 1.4 solar radius and 1.2 solar masses. When the star's surface is not covered by dark spots, its Teff is 6000K. However, when 40% of its surface is covered by spots, the overall photospheric effective temperature decreases to 5500 K. In the stellar photosphere, astronomers have observed that the ratio of ionized to neutral atoms of Mg decreases when the star has spots. What is the factor by which this ratio changes when the star does not have spots compared to when it has spots? Assume the energy of ionization corresponds to a wavelength of approximately 1448 Å, and that the stellar photosphere is in LTE.\\n\",\n \"Answer Choices:\\n\",\n \"(A) ~ 3.9\\n\",\n \"(B) ~ 5.2\\n\",\n \"(C) ~ 8.2\\n\",\n \"(D) ~ 6.3\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 3303ceee5a75c570\\n\",\n \"## source_type: Idavidrein/gpqa\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"A spin-half particle is in a mixed ensemble. The spin half system is in state |\\\\psi1 \\\\rangle with probabilty (1/3) and in state in state |\\\\psi2 \\\\rangle with probability (2/3).\\n\",\n \" Here |\\\\psi1 \\\\rangle is a state in linear superposition (1/2)|\\\\uparrow\\\\rangle+(\\\\sqrt(3)/2) |\\\\downarrow\\\\rangle of its spin-up and spin-down states and \\n\",\n \"|\\\\psi2 \\\\rangle is linear superposition (1/\\\\sqrt(2)|\\\\uparrow\\\\rangle+(1/\\\\sqrt(2)) |\\\\downarrow\\\\rangle of its spin-up and spin-down states. \\n\",\n \"If |\\\\uparrow\\\\rangle and |\\\\downarrow\\\\rangle are the eigenstates of \\\\sigma{z} , then what is the expectation value up to two decimal place, of the operator 10\\\\sigma{z}+5\\\\sigma_{x} ? Here, symbols have their usual meanings\\n\",\n \"Answer Choices:\\n\",\n \"(A) 3.86\\n\",\n \"(B) 1.24\\n\",\n \"(C) 4.62\\n\",\n \"(D) 5.28\\n\",\n \"\\n\",\n \"\\n\",\n \"----------------------------------------------------------------------------------------------------\\n\",\n \"## Question hash: 39e92361c025735f\\n\",\n \"## source_type: qq8933/AIME_1983_2024\\n\",\n \"## Question statement\\n\",\n \"\\n\",\n \"Triangles $\\\\triangle ABC$ and $\\\\triangle A'B'C'$ lie in the coordinate plane with vertices $A(0,0)$ , $B(0,12)$ , $C(16,0)$ , $A'(24,18)$ , $B'(36,18)$ , $C'(24,2)$ . A rotation of $m$ degrees clockwise around the point $(x,y)$ where $0
\n![\"logo\"](\"assets/logo.png\")
\n
\n\n--------------------------------------------------------------------------------\n\n| [**Blog**](https://lmsys.org/blog/2024-01-17-sglang/) | [**Paper**](https://arxiv.org/abs/2312.07104) |\n\nSGLang is a structured generation language designed for large language models (LLMs).\nIt makes your interaction with LLMs faster and more controllable by co-designing the frontend language and the runtime system.\n\nThe core features of SGLang include:\n- **A Flexible Front-End Language**: This allows for easy programming of LLM applications with multiple chained generation calls, advanced prompting techniques, control flow, multiple modalities, parallelism, and external interaction.\n- **A High-Performance Runtime with RadixAttention**: This feature significantly accelerates the execution of complex LLM programs by automatic KV cache reuse across multiple calls. It also supports other common techniques like continuous batching and tensor parallelism.\n\n## News\n- [2024/02] 🔥 SGLang enables **3x faster JSON decoding** with compressed finite state machine ([blog](https://lmsys.org/blog/2024-02-05-compressed-fsm/)).\n- [2024/01] 🔥 SGLang powers the serving of the official **LLaVA v1.6** release demo ([usage](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#demo)).\n- [2024/01] SGLang provides up to **5x faster inference** with RadixAttention ([blog](https://lmsys.org/blog/2024-01-17-sglang/)).\n\n## Contents\n- [Install](#install)\n- [Quick Start](#quick-start)\n- [Frontend: Structured Generation Language (SGLang)](#frontend-structured-generation-language-sglang)\n- [Backend: SGLang Runtime (SRT)](#backend-sglang-runtime-srt)\n- [Benchmark And Performance](#benchmark-and-performance)\n- [Roadmap](#roadmap)\n- [Citation And Acknowledgment](#citation-and-acknowledgment)\n\n## Install\n\n### Method 1: With pip\n```\npip install \"sglang[all]\"\n```\n\n### Method 2: From source\n```\ngit clone git@github.com:sgl-project/sglang.git\ncd sglang\n\npip install --upgrade pip\npip install -e \"python[all]\"\n```\n\n### Notes\n- If you are using older GPUs (NVIDIA V100, T4), please pick the correct triton compiler version to avoid some known bugs.\n - For NVIDIA T4, please use `pip install \"triton>=2.2.0\"`.\n - For NVIDIA V100, please install the [nightly](https://triton-lang.org/main/getting-started/installation.html) version.\n- If you only need to use the OpenAI backend, you can avoid installing other dependencies by using `pip install \"sglang[openai]\"`\n\n\n## Quick Start\nThe example below shows how to use sglang to answer a mulit-turn question.\n\n### Using Local Models\nFirst, launch a server with\n```\npython -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000\n```\n\nThen, connect to the server and answer a multi-turn question.\n\n```python\nfrom sglang import function, system, user, assistant, gen, set_default_backend, RuntimeEndpoint\n\n@function\ndef multi_turn_question(s, question_1, question_2):\n s += system(\"You are a helpful assistant.\")\n s += user(question_1)\n s += assistant(gen(\"answer_1\", max_tokens=256))\n s += user(question_2)\n s += assistant(gen(\"answer_2\", max_tokens=256))\n\nset_default_backend(RuntimeEndpoint(\"http://localhost:30000\"))\n\nstate = multi_turn_question.run(\n question_1=\"What is the capital of the United States?\",\n question_2=\"List two local attractions.\",\n)\n\nfor m in state.messages():\n print(m[\"role\"], \":\", m[\"content\"])\n\nprint(state[\"answer_1\"])\n```\n\n### Using OpenAI Models\nSet the OpenAI API Key\n```\nexport OPENAI_API_KEY=sk-******\n```\n\nThen, answer a multi-turn question.\n```python\nfrom sglang import function, system, user, assistant, gen, set_default_backend, OpenAI\n\n@function\ndef multi_turn_question(s, question_1, question_2):\n s += system(\"You are a helpful assistant.\")\n s += user(question_1)\n s += assistant(gen(\"answer_1\", max_tokens=256))\n s += user(question_2)\n s += assistant(gen(\"answer_2\", max_tokens=256))\n\nset_default_backend(OpenAI(\"gpt-3.5-turbo\"))\n\nstate = multi_turn_question.run(\n question_1=\"What is the capital of the United States?\",\n question_2=\"List two local attractions.\",\n)\n\nfor m in state.messages():\n print(m[\"role\"], \":\", m[\"content\"])\n\nprint(state[\"answer_1\"])\n```\n\n### More Examples\n\nAnthropic and VertexAI (Gemini) models are also supported.\nYou can find more examples at [examples/quick_start](examples/quick_start).\n\n## Frontend: Structured Generation Language (SGLang)\n\nTo begin with, import sglang.\n```python\nimport sglang as sgl\n```\n\n`sglang` provides some simple primitives such as `gen`, `select`, `fork`, `image`.\nYou can implement your prompt flow in a function decorated by `sgl.function`.\nYou can then invoke the function with `run` or `run_batch`.\nThe system will manage the state, chat template, parallelism and batching for you.\n\nThe complete code for the examples below can be found at [readme_examples.py](examples/usage/readme_examples.py)\n\n### Control Flow\nYou can use any Python code within the function body, including control flow, nested function calls, and external libraries.\n\n```python\n@sgl.function\ndef tool_use(s, question):\n s += \"To answer this question: \" + question + \". \"\n s += \"I need to use a \" + sgl.gen(\"tool\", choices=[\"calculator\", \"search engine\"]) + \". \"\n\n if s[\"tool\"] == \"calculator\":\n s += \"The math expression is\" + sgl.gen(\"expression\")\n elif s[\"tool\"] == \"search engine\":\n s += \"The key word to search is\" + sgl.gen(\"word\")\n```\n\n### Parallelism\nUse `fork` to launch parallel prompts.\nBecause `sgl.gen` is non-blocking, the for loop below issues two generation calls in parallel.\n\n```python\n@sgl.function\ndef tip_suggestion(s):\n s += (\n \"Here are two tips for staying healthy: \"\n \"1. Balanced Diet. 2. Regular Exercise.\\n\\n\"\n )\n\n forks = s.fork(2)\n for i, f in enumerate(forks):\n f += f\"Now, expand tip {i+1} into a paragraph:\\n\"\n f += sgl.gen(f\"detailed_tip\", max_tokens=256, stop=\"\\n\\n\")\n\n s += \"Tip 1:\" + forks[0][\"detailed_tip\"] + \"\\n\"\n s += \"Tip 2:\" + forks[1][\"detailed_tip\"] + \"\\n\"\n s += \"In summary\" + sgl.gen(\"summary\")\n```\n\n### Multi Modality\nUse `sgl.image` to pass an image as input.\n\n```python\n@sgl.function\ndef image_qa(s, image_file, question):\n s += sgl.user(sgl.image(image_file) + question)\n s += sgl.assistant(sgl.gen(\"answer\", max_tokens=256)\n```\n\nSee also [srt_example_llava.py](examples/quick_start/srt_example_llava.py).\n\n### Constrained Decoding\nUse `regex` to specify a regular expression as a decoding constraint.\nThis is only supported for local models.\n\n```python\n@sgl.function\ndef regular_expression_gen(s):\n s += \"Q: What is the IP address of the Google DNS servers?\\n\"\n s += \"A: \" + sgl.gen(\n \"answer\",\n temperature=0,\n regex=r\"((25[0-5]|2[0-4]\\d|[01]?\\d\\d?).){3}(25[0-5]|2[0-4]\\d|[01]?\\d\\d?)\",\n )\n```\n\n### JSON Decoding\nUse `regex` to specify a JSON schema with a regular expression.\n\n```python\ncharacter_regex = (\n r\"\"\"\\{\\n\"\"\"\n + r\"\"\" \"name\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"house\": \"(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)\",\\n\"\"\"\n + r\"\"\" \"blood status\": \"(Pure-blood|Half-blood|Muggle-born)\",\\n\"\"\"\n + r\"\"\" \"occupation\": \"(student|teacher|auror|ministry of magic|death eater|order of the phoenix)\",\\n\"\"\"\n + r\"\"\" \"wand\": \\{\\n\"\"\"\n + r\"\"\" \"wood\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"core\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"length\": [0-9]{1,2}\\.[0-9]{0,2}\\n\"\"\"\n + r\"\"\" \\},\\n\"\"\"\n + r\"\"\" \"alive\": \"(Alive|Deceased)\",\\n\"\"\"\n + r\"\"\" \"patronus\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"bogart\": \"[\\w\\d\\s]{1,16}\"\\n\"\"\"\n + r\"\"\"\\}\"\"\"\n)\n\n@sgl.function\ndef character_gen(s, name):\n s += name + \" is a character in Harry Potter. Please fill in the following information about this character.\\n\"\n s += sgl.gen(\"json_output\", max_tokens=256, regex=character_regex)\n```\n\nSee also [json_decode.py](examples/usage/json_decode.py) for an additional example on specifying formats with Pydantic models.\n\n\n### Batching\nUse `run_batch` to run a batch of requests with continuous batching.\n\n```python\n@sgl.function\ndef text_qa(s, question):\n s += \"Q: \" + question + \"\\n\"\n s += \"A:\" + sgl.gen(\"answer\", stop=\"\\n\")\n\nstates = text_qa.run_batch(\n [\n {\"question\": \"What is the capital of the United Kingdom?\"},\n {\"question\": \"What is the capital of France?\"},\n {\"question\": \"What is the capital of Japan?\"},\n ],\n progress_bar=True\n)\n```\n\n### Streaming\nAdd `stream=True` to enable streaming.\n\n```python\n@sgl.function\ndef text_qa(s, question):\n s += \"Q: \" + question + \"\\n\"\n s += \"A:\" + sgl.gen(\"answer\", stop=\"\\n\")\n\nstate = text_qa.run(\n question=\"What is the capital of France?\",\n temperature=0.1,\n stream=True\n)\n\nfor out in state.text_iter():\n print(out, end=\"\", flush=True)\n```\n\n### Tips and Implementation Details\n- The `choices` argument in `sgl.gen` is implemented by computing the normalized log probabilities of all choices and selecting the one with the highest probability.\n- The `regex` argument in `sgl.gen` is implemented through autoregressive decoding with logit bias masking, according to the constraints set by the regex.\n\n## Backend: SGLang Runtime (SRT)\nThe SGLang Runtime (SRT) is designed to work best with the SGLang frontend.\nHowever, it can also be used as a standalone API server.\nIn this case, the [RadixAttention](https://arxiv.org/abs/2312.07104) can still greatly accelerate many use cases with automatic KV cache reuse.\n\n### Usage\nLaunch a server\n```\npython -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000\n```\n\nSend a request\n```\ncurl http://localhost:30000/generate \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"text\": \"Once upon a time,\",\n \"sampling_params\": {\n \"max_new_tokens\": 16,\n \"temperature\": 0\n }\n }'\n```\nLearn more about the argument format [here](docs/sampling_params.md).\n\n### OpenAI Compatible API\n\nIn addition, the server supports an experimental OpenAI-compatible API.\n\n```python\nimport openai\nclient = openai.Client(\n base_url=\"http://127.0.0.1:30000/v1\", api_key=\"EMPTY\")\n\n# Text completion\nresponse = client.completions.create(\n\tmodel=\"default\",\n\tprompt=\"The capital of France is\",\n\ttemperature=0,\n\tmax_tokens=32,\n)\nprint(response)\n\n# Chat completion\nresponse = client.chat.completions.create(\n model=\"default\",\n messages=[\n {\"role\": \"system\", \"content\": \"You are a helpful AI assistant\"},\n {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n ],\n temperature=0,\n max_tokens=64,\n)\nprint(response)\n```\n\nIn above example, the server uses the chat template specified in the model tokenizer.\nYou can override the chat template if needed when launching the server:\n\n```\npython -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --chat-template llama-2\n```\n\nIf the chat template you are looking for is missing, you are welcome to contribute it.\nMeanwhile, you can also temporary register your chat template as follows:\n\n```json\n{\n \"name\": \"my_model\",\n \"system\": \"<|im_start|>system\",\n \"user\": \"<|im_start|>user\",\n \"assistant\": \"<|im_start|>assistant\",\n \"sep_style\": \"CHATML\",\n \"sep\": \"<|im_end|>\",\n \"stop_str\": [\"<|im_end|>\", \"<|im_start|>\"]\n}\n```\n\n```\npython -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --chat-template ./my_model_template.json\n```\n\n### Additional Arguments\n- Add `--tp 2` to enable tensor parallelism.\n```\npython -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --tp 2\n```\n- If you see out-of-memory errors during serving, please try to reduce the memory usage of the KV cache pool by setting a smaller value of `--mem-fraction-static`. The default value is `0.9`\n```\npython -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --mem-fraction-static 0.7\n```\n- You can turn on [flashinfer](docs/flashinfer.md) to acclerate the inference by using highly optimized CUDA kernels.\n\n### Supported Models\n- Llama\n- Mistral\n- Mixtral\n- Qwen / Qwen 2\n- Gemma\n - Please add a new flag `--attention-reduce-in-fp32` to avoid some precision errors.\n - `python -m sglang.launch_server --model-path google/gemma-7b-it --port 30000 --attention-reduce-in-fp32`\n- LLaVA\n - `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.5-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --chat-template vicuna_v1.1 --port 30000`\n - `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.6-vicuna-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --chat-template vicuna_v1.1 --port 30000`\n - `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.6-34b --tokenizer-path liuhaotian/llava-v1.6-34b-tokenizer --port 3000`\n- Yi-VL\n - see [srt_example_yi_vl.py](examples/quick_start/srt_example_yi_vl.py).\n- AWQ/GPTQ quantization\n\n## Benchmark And Performance\n\n- Llama-7B on NVIDIA A10G, FP16, Tensor Parallelism=1\n\n\n- Mixtral-8x7B on NVIDIA A10G, FP16, Tensor Parallelism=8\n\n\nLearn more [here](docs/benchmark_results.md).\n\n## Roadmap\nhttps://github.com/sgl-project/sglang/issues/157\n\n## Citation And Acknowledgment\n```\n@misc{zheng2023efficiently,\n title={Efficiently Programming Large Language Models using SGLang},\n author={Lianmin Zheng and Liangsheng Yin and Zhiqiang Xie and Jeff Huang and Chuyue Sun and Cody Hao Yu and Shiyi Cao and Christos Kozyrakis and Ion Stoica and Joseph E. Gonzalez and Clark Barrett and Ying Sheng},\n year={2023},\n eprint={2312.07104},\n archivePrefix={arXiv},\n primaryClass={cs.AI}\n}\n```\n\n[](https://huggingface.co/papers/2312.07104)\n\n\nWe learned from the design and reused some code of the following projects: [Guidance](https://github.com/guidance-ai/guidance), [vLLM](https://github.com/vllm-project/vllm), [LightLLM](https://github.com/ModelTC/lightllm), [FlashInfer](https://github.com/flashinfer-ai/flashinfer), [Outlines](https://github.com/outlines-dev/outlines), [LMQL](https://github.com/eth-sri/lmql).\n"
},
{
"path": "eval/rebase/sglang/benchmark/dspy/README.md",
"content": "## Install\n\n```\npip3 install dspy-ai\n```\n\nTurn off cache at https://github.com/stanfordnlp/dspy/blob/34d8420383ec752037aa271825c1d3bf391e1277/dsp/modules/cache_utils.py#L10.\n```\ncache_turn_on = False\n```\n\n## Benchmark SGLang\n```\npython -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000\n```\n\n```\npython3 bench_dspy_intro.py --backend sglang\n```\n\n\n## Benchmark TGI\n```\ndocker run --name tgi --rm -ti --gpus all --network host \\\n -v /home/ubuntu/model_weights/Llama-2-7b-chat-hf:/Llama-2-7b-chat-hf \\\n ghcr.io/huggingface/text-generation-inference:1.3.0 \\\n --model-id /Llama-2-7b-chat-hf --num-shard 1 --trust-remote-code \\\n --max-input-length 2048 --max-total-tokens 4096 \\\n --port 24000\n```\n\n```\npython3 bench_dspy_intro.py --backend tgi\n```\n\n\n\n## Benchmark vLLM\n```\npython3 -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000\n```\n\n```\npython3 bench_dspy_intro.py --backend vllm\n```\n"
},
{
"path": "eval/rebase/sglang/benchmark/dspy/bench_dspy_intro.py",
"content": "\"\"\"\nAdapted from\nhttps://github.com/stanfordnlp/dspy/blob/34d8420383ec752037aa271825c1d3bf391e1277/intro.ipynb#L9\n\"\"\"\nimport argparse\n\nimport dspy\nfrom dspy.datasets import HotPotQA\n\n\nclass BasicQA(dspy.Signature):\n \"\"\"Answer questions with short factoid answers.\"\"\"\n\n question = dspy.InputField()\n answer = dspy.OutputField(desc=\"often between 1 and 5 words\")\n\n\nclass GenerateAnswer(dspy.Signature):\n \"\"\"Answer questions with short factoid answers.\"\"\"\n\n context = dspy.InputField(desc=\"may contain relevant facts\")\n question = dspy.InputField()\n answer = dspy.OutputField(desc=\"often between 1 and 5 words\")\n\n\nclass RAG(dspy.Module):\n def __init__(self, num_passages=3):\n super().__init__()\n\n self.retrieve = dspy.Retrieve(k=num_passages)\n self.generate_answer = dspy.ChainOfThought(GenerateAnswer)\n \n def forward(self, question):\n context = self.retrieve(question).passages\n prediction = self.generate_answer(context=context, question=question)\n return dspy.Prediction(context=context, answer=prediction.answer)\n\n\ndef main(args):\n #lm = dspy.OpenAI(model='gpt-3.5-turbo')\n if args.backend == \"tgi\":\n lm = dspy.HFClientTGI(model=\"meta-llama/Llama-2-7b-chat-hf\", port=args.port,\n url=\"http://localhost\")\n elif args.backend == \"sglang\":\n lm = dspy.HFClientSGLang(model=\"meta-llama/Llama-2-7b-chat-hf\", port=args.port,\n url=\"http://localhost\")\n elif args.backend == \"vllm\":\n lm = dspy.HFClientVLLM(model=\"meta-llama/Llama-2-7b-chat-hf\", port=args.port,\n url=\"http://localhost\")\n else:\n raise ValueError(f\"Invalid backend: {args.backend}\")\n\n colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n dspy.settings.configure(lm=lm, rm=colbertv2_wiki17_abstracts)\n\n # Load the dataset.\n dataset = HotPotQA(train_seed=1, train_size=20, eval_seed=2023, dev_size=args.dev_size,\n test_size=0)\n\n # Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.\n trainset = [x.with_inputs('question') for x in dataset.train]\n devset = [x.with_inputs('question') for x in dataset.dev]\n\n print(len(trainset), len(devset))\n\n train_example = trainset[0]\n print(f\"Question: {train_example.question}\")\n print(f\"Answer: {train_example.answer}\")\n\n dev_example = devset[18]\n print(f\"Question: {dev_example.question}\")\n print(f\"Answer: {dev_example.answer}\")\n print(f\"Relevant Wikipedia Titles: {dev_example.gold_titles}\")\n\n print(f\"For this dataset, training examples have input keys {train_example.inputs().keys()} and label keys {train_example.labels().keys()}\")\n print(f\"For this dataset, dev examples have input keys {dev_example.inputs().keys()} and label keys {dev_example.labels().keys()}\")\n\n # Define the predictor.\n generate_answer = dspy.Predict(BasicQA)\n \n # Call the predictor on a particular input.\n pred = generate_answer(question=dev_example.question)\n \n # Print the input and the prediction.\n print(f\"Question: {dev_example.question}\")\n print(f\"Predicted Answer: {pred.answer}\")\n\n lm.inspect_history(n=1)\n\n # Define the predictor. Notice we're just changing the class. The signature BasicQA is unchanged.\n generate_answer_with_chain_of_thought = dspy.ChainOfThought(BasicQA)\n \n # Call the predictor on the same input.\n pred = generate_answer_with_chain_of_thought(question=dev_example.question)\n \n # Print the input, the chain of thought, and the prediction.\n print(f\"Question: {dev_example.question}\")\n print(f\"Thought: {pred.rationale.split('.', 1)[1].strip()}\")\n print(f\"Predicted Answer: {pred.answer}\")\n\n retrieve = dspy.Retrieve(k=3)\n topK_passages = retrieve(dev_example.question).passages\n\n print(f\"Top {retrieve.k} passages for question: {dev_example.question} \\n\", '-' * 30, '\\n')\n\n for idx, passage in enumerate(topK_passages):\n print(f'{idx+1}]', passage, '\\n')\n\n retrieve(\"When was the first FIFA World Cup held?\").passages[0]\n\n from dspy.teleprompt import BootstrapFewShot\n \n # Validation logic: check that the predicted answer is correct.\n # Also check that the retrieved context does actually contain that answer.\n def validate_context_and_answer(example, pred, trace=None):\n answer_EM = dspy.evaluate.answer_exact_match(example, pred)\n answer_PM = dspy.evaluate.answer_passage_match(example, pred)\n return answer_EM and answer_PM\n \n # Set up a basic teleprompter, which will compile our RAG program.\n teleprompter = BootstrapFewShot(metric=validate_context_and_answer)\n\n # Compile!\n compiled_rag = teleprompter.compile(RAG(), trainset=trainset)\n\n # Ask any question you like to this simple RAG program.\n my_question = \"What castle did David Gregory inherit?\"\n \n # Get the prediction. This contains `pred.context` and `pred.answer`.\n pred = compiled_rag(my_question)\n \n # Print the contexts and the answer.\n print(f\"Question: {my_question}\")\n print(f\"Predicted Answer: {pred.answer}\")\n print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}\")\n\n from dspy.evaluate.evaluate import Evaluate\n\n # Set up the `evaluate_on_hotpotqa` function. We'll use this many times below.\n evaluate_on_hotpotqa = Evaluate(devset=devset, num_threads=args.num_threads, display_progress=True, display_table=5)\n \n # Evaluate the `compiled_rag` program with the `answer_exact_match` metric.\n metric = dspy.evaluate.answer_exact_match\n evaluate_on_hotpotqa(compiled_rag, metric=metric)\n \n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--port\", type=int)\n parser.add_argument(\"--num-threads\", type=int, default=32)\n parser.add_argument(\"--dev-size\", type=int, default=150)\n parser.add_argument(\"--backend\", type=str, choices=[\"sglang\", \"tgi\", \"vllm\"],\n default=\"sglang\")\n args = parser.parse_args()\n\n if args.port is None:\n default_port = {\n \"vllm\": 21000,\n \"lightllm\": 22000,\n \"tgi\": 24000,\n \"sglang\": 30000,\n }\n args.port = default_port.get(args.backend, None)\n\n main(args)\n"
},
{
"path": "eval/rebase/sglang/benchmark/generative_agents/README.md",
"content": "## Download the dataset\n\n```\nwget -O agent_calls.jsonl https://drive.google.com/uc?export=download&id=19qLpD45e9JGTKF2cUjJJegwzSUEZEKht\n```\n\n## Run benchmark\n\nEnsure that this benchmark is run in a serial manner (using --parallel 1) to preserve any potential dependencies between requests.\n\n### Benchmark sglang\n```\npython -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000\n```\n\n```\npython3 bench_sglang.py --num-events 1000 --parallel 1\n```\n\n### Benchmark vllm\n```\npython3 -m vllm.entrypoints.api_server --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000\n```\n\n```\npython3 bench_other.py --num-events 1000 --backend vllm --parallel 1\n```\n\n### Benchmark guidance\n```\npython3 bench_other.py --num-events 1000 --backend guidance --parallel 1\n```\n"
},
{
"path": "eval/rebase/sglang/benchmark/generative_agents/agent_functions.py",
"content": "import sglang as sgl\n\n# here are the top five agent functions contributing ~70% LLM calls\n# reference: https://github.com/joonspk-research/generative_agents/\n\n\n@sgl.function\ndef poignancy_event(s, persona_name, persona_iss, event):\n s += \"Here is a brief description of \" + persona_name + \".\\n\"\n s += persona_iss + \"\\n\"\n s += \"On the scale of 1 to 10, where 1 is purely mundane (e.g., brushing teeth, making bed) and 10 is extremely poignant (e.g., a break up, college acceptance), rate the likely poignancy of the following event for\"\n s += persona_name + \".\\n\\n\"\n s += \"Event: \" + event\n s += \"Rate (return a number between 1 to 10):\"\n s += sgl.gen(name=\"Rate\", max_tokens=2)\n\n\ndef poignancy_event_prompt(persona_name, persona_iss, event):\n # return prompt and max_tokens\n s = \"\"\n s += \"Here is a brief description of \" + persona_name + \".\\n\"\n s += persona_iss + \"\\n\"\n s += \"On the scale of 1 to 10, where 1 is purely mundane (e.g., brushing teeth, making bed) and 10 is extremely poignant (e.g., a break up, college acceptance), rate the likely poignancy of the following event for\"\n s += persona_name + \".\\n\\n\"\n s += \"Event: \" + event\n s += \"Rate (return a number between 1 to 10):\"\n return {\"prompt\": s, \"max_tokens\": 2, \"stop\": None}\n\n\n@sgl.function\ndef generate_event_triple(s, persona_name, action):\n s += \"\"\"Task: Turn the input into (subject, predicate, object).\nInput: Sam Johnson is eating breakfast. \nOutput: (Dolores Murphy, eat, breakfast) \n--- \nInput: Joon Park is brewing coffee.\nOutput: (Joon Park, brew, coffee)\n---\nInput: Jane Cook is sleeping. \nOutput: (Jane Cook, is, sleep)\n---\nInput: Michael Bernstein is writing email on a computer. \nOutput: (Michael Bernstein, write, email)\n---\nInput: Percy Liang is teaching students in a classroom. \nOutput: (Percy Liang, teach, students)\n---\nInput: Merrie Morris is running on a treadmill. \nOutput: (Merrie Morris, run, treadmill)\n---\"\"\"\n s += persona_name + \"is\" + action + \".\\n\"\n s += \"(\" + persona_name + \",\"\n s += sgl.gen(name=\"Triple\", max_tokens=20, stop=\")\")\n\n\ndef generate_event_triple_prompt(persona_name, action):\n s = \"\"\n s += \"\"\"Task: Turn the input into (subject, predicate, object).\nInput: Sam Johnson is eating breakfast. \nOutput: (Dolores Murphy, eat, breakfast) \n--- \nInput: Joon Park is brewing coffee.\nOutput: (Joon Park, brew, coffee)\n---\nInput: Jane Cook is sleeping. \nOutput: (Jane Cook, is, sleep)\n---\nInput: Michael Bernstein is writing email on a computer. \nOutput: (Michael Bernstein, write, email)\n---\nInput: Percy Liang is teaching students in a classroom. \nOutput: (Percy Liang, teach, students)\n---\nInput: Merrie Morris is running on a treadmill. \nOutput: (Merrie Morris, run, treadmill)\n---\"\"\"\n s += persona_name + \"is\" + action + \".\\n\"\n s += \"(\" + persona_name + \",\"\n return {\"prompt\": s, \"max_tokens\": 20, \"stop\": \")\"}\n\n\n@sgl.function\ndef generate_pronunciatio(s, action):\n s += \"Convert an action description to an emoji (important: use two or less emojis).\\n\"\n s += \"Action description: \" + action + \".\\n\"\n s += \"Emoji:\" + sgl.gen(name=\"Emoji\", max_tokens=6)\n\n\ndef generate_pronunciatio_prompt(action):\n s = \"\"\n s += \"Convert an action description to an emoji (important: use two or less emojis).\\n\"\n s += \"Action description: \" + action + \".\\n\"\n s += \"Emoji:\"\n return {\"prompt\": s, \"max_tokens\": 6, \"stop\": None}\n\n\n@sgl.function\ndef action_location_sector(\n s,\n persona_name,\n living_sector,\n living_sector_areas,\n current_sector,\n current_sector_areas,\n daily_plan,\n sector_options,\n current_action,\n next_action,\n):\n s += \"\"\"Task -- choose an appropriate area from the area options for a task at hand. \nSam Kim lives in {Sam Kim's house} that has Sam Kim's room, bathroom, kitchen.\nSam Kim is currently in {Sam Kim's house} that has Sam Kim's room, bathroom, kitchen. \nArea options: {Sam Kim's house, The Rose and Crown Pub, Hobbs Cafe, Oak Hill College, Johnson Park, Harvey Oak Supply Store, The Willows Market and Pharmacy}.\n* Stay in the current area if the activity can be done there. Only go out if the activity needs to take place in another place.\n* Must be one of the \"Area options,\" verbatim.\nFor taking a walk, Sam Kim should go to the following area: {Johnson Park}\n---\nJane Anderson lives in {Oak Hill College Student Dormatory} that has Jane Anderson's room.\nJane Anderson is currently in {Oak Hill College} that has a classroom, library\nArea options: {Oak Hill College Student Dormatory, The Rose and Crown Pub, Hobbs Cafe, Oak Hill College, Johnson Park, Harvey Oak Supply Store, The Willows Market and Pharmacy}. \n* Stay in the current area if the activity can be done there. Only go out if the activity needs to take place in another place.\n* Must be one of the \"Area options,\" verbatim.\nFor eating dinner, Jane Anderson should go to the following area: {Hobbs Cafe}\n---\"\"\"\n s += (persona_name + \" lives in \" + living_sector + \" that has \" +\n living_sector_areas + \".\\n\")\n s += (persona_name + \" is currently in \" + current_sector + \" that has \" +\n current_sector_areas + \".\\n\")\n s += daily_plan + \".\\n\"\n s += \"Area options: \" + sector_options + \".\\n\"\n s += \"\"\"* Stay in the current area if the activity can be done there. Only go out if the activity needs to take place in another place.\n* Must be one of the \"Area options,\" verbatim.\\n\"\"\"\n s += (persona_name + \" is \" + current_action + \". For \" + next_action +\n \", \" + persona_name + \" should go to the following area: {\")\n s += sgl.gen(name=\"Location\", max_tokens=10, stop=\"}\")\n\n\ndef action_location_sector_prompt(\n persona_name,\n living_sector,\n living_sector_areas,\n current_sector,\n current_sector_areas,\n daily_plan,\n sector_options,\n current_action,\n next_action,\n):\n s = \"\"\n s += \"\"\"Task -- choose an appropriate area from the area options for a task at hand. \nSam Kim lives in {Sam Kim's house} that has Sam Kim's room, bathroom, kitchen.\nSam Kim is currently in {Sam Kim's house} that has Sam Kim's room, bathroom, kitchen. \nArea options: {Sam Kim's house, The Rose and Crown Pub, Hobbs Cafe, Oak Hill College, Johnson Park, Harvey Oak Supply Store, The Willows Market and Pharmacy}.\n* Stay in the current area if the activity can be done there. Only go out if the activity needs to take place in another place.\n* Must be one of the \"Area options,\" verbatim.\nFor taking a walk, Sam Kim should go to the following area: {Johnson Park}\n---\nJane Anderson lives in {Oak Hill College Student Dormatory} that has Jane Anderson's room.\nJane Anderson is currently in {Oak Hill College} that has a classroom, library\nArea options: {Oak Hill College Student Dormatory, The Rose and Crown Pub, Hobbs Cafe, Oak Hill College, Johnson Park, Harvey Oak Supply Store, The Willows Market and Pharmacy}. \n* Stay in the current area if the activity can be done there. Only go out if the activity needs to take place in another place.\n* Must be one of the \"Area options,\" verbatim.\nFor eating dinner, Jane Anderson should go to the following area: {Hobbs Cafe}\n---\"\"\"\n s += (persona_name + \" lives in \" + living_sector + \" that has \" +\n living_sector_areas + \".\\n\")\n s += (persona_name + \" is currently in \" + current_sector + \" that has \" +\n current_sector_areas + \".\\n\")\n s += daily_plan + \".\\n\"\n s += \"Area options: \" + sector_options + \".\\n\"\n s += \"\"\"* Stay in the current area if the activity can be done there. Only go out if the activity needs to take place in another place.\n* Must be one of the \"Area options,\" verbatim.\\n\"\"\"\n s += (persona_name + \" is \" + current_action + \". For \" + next_action +\n \", \" + persona_name + \" should go to the following area: {\")\n return {\"prompt\": s, \"max_tokens\": 10, \"stop\": \"}\"}\n\n\n@sgl.function\ndef action_location_object(s, persona_name, target_sector, target_sector_areas,\n current_action, next_action):\n s += \"\"\"\nJane Anderson is in kitchen in Jane Anderson's house.\nJane Anderson is going to Jane Anderson's house that has the following areas: {kitchen, bedroom, bathroom}\nStay in the current area if the activity can be done there. Never go into other people's rooms unless necessary.\nFor cooking, Jane Anderson should go to the following area in Jane Anderson's house:\nAnswer: {kitchen}\n---\nTom Watson is in common room in Tom Watson's apartment. \nTom Watson is going to Hobbs Cafe that has the following areas: {cafe}\nStay in the current area if the activity can be done there. Never go into other people's rooms unless necessary.\nFor getting coffee, Tom Watson should go to the following area in Hobbs Cafe:\nAnswer: {cafe}\n---\"\"\"\n s += (persona_name + \" is going to \" + target_sector +\n \" that has the following areas: {\" + target_sector_areas + \"}\\n\")\n s += \"\"\"* Stay in the current area if the activity can be done there. \n* NEVER go into other people's rooms unless necessary.\"\"\"\n s += (persona_name + \" is \" + current_action + \". For \" + next_action +\n \", \" + persona_name + \"should go to the following area in \" +\n target_sector)\n s += \" (MUST pick one of {\" + target_sector_areas + \"}):\\n\"\n s += \"Answer: {\" + sgl.gen(name=\"Area\", max_tokens=5, stop=\"}\")\n\n\ndef action_location_object_prompt(persona_name, target_sector,\n target_sector_areas, current_action,\n next_action):\n s = \"\"\n s += \"\"\"\nJane Anderson is in kitchen in Jane Anderson's house.\nJane Anderson is going to Jane Anderson's house that has the following areas: {kitchen, bedroom, bathroom}\nStay in the current area if the activity can be done there. Never go into other people's rooms unless necessary.\nFor cooking, Jane Anderson should go to the following area in Jane Anderson's house:\nAnswer: {kitchen}\n---\nTom Watson is in common room in Tom Watson's apartment. \nTom Watson is going to Hobbs Cafe that has the following areas: {cafe}\nStay in the current area if the activity can be done there. Never go into other people's rooms unless necessary.\nFor getting coffee, Tom Watson should go to the following area in Hobbs Cafe:\nAnswer: {cafe}\n---\"\"\"\n s += (persona_name + \" is going to \" + target_sector +\n \" that has the following areas: {\" + target_sector_areas + \"}\\n\")\n s += \"\"\"* Stay in the current area if the activity can be done there. \n* NEVER go into other people's rooms unless necessary.\"\"\"\n s += (persona_name + \" is \" + current_action + \". For \" + next_action +\n \", \" + persona_name + \"should go to the following area in \" +\n target_sector)\n s += \" (MUST pick one of {\" + target_sector_areas + \"}):\\n\"\n s += \"Answer: {\"\n return {\"prompt\": s, \"max_tokens\": 5, \"stop\": \"}\"}\n"
},
{
"path": "eval/rebase/sglang/benchmark/generative_agents/bench_other.py",
"content": "import argparse\nfrom functools import partial\nimport json\nimport time\nfrom pathlib import Path\n\nfrom tqdm import tqdm\nfrom sglang.test.test_utils import (\n add_common_other_args_and_parse,\n call_generate_lightllm,\n call_generate_vllm,\n call_generate_srt_raw,\n)\nfrom sglang.utils import read_jsonl, dump_state_text\n\nfrom agent_functions import (\n poignancy_event_prompt,\n generate_event_triple_prompt,\n generate_pronunciatio_prompt,\n action_location_sector_prompt,\n action_location_object_prompt,\n)\n\n\ndef main(args):\n lines = read_jsonl(args.data_path)[:args.num_events]\n mapping = {\n \"poignancy_event\": poignancy_event_prompt,\n \"generate_event_triple\": generate_event_triple_prompt,\n \"generate_pronunciatio\": generate_pronunciatio_prompt,\n \"action_location_sector\": action_location_sector_prompt,\n \"action_location_object\": action_location_object_prompt,\n }\n\n arguments = [mapping[k](**v) for l in lines for k, v in l.items()]\n states = []\n\n # Select backend\n if args.backend == \"lightllm\":\n url = f\"{args.host}:{args.port}/generate\"\n call_generate = partial(call_generate_lightllm, url=url)\n elif args.backend == \"vllm\":\n url = f\"{args.host}:{args.port}/generate\"\n call_generate = partial(call_generate_vllm, url=url)\n elif args.backend == \"srt-raw\":\n url = f\"{args.host}:{args.port}/generate\"\n call_generate = partial(call_generate_srt_raw, url=url)\n elif args.backend == \"guidance\":\n from guidance import models, gen\n\n model = models.LlamaCpp(\n str(Path.home()) + \"/model_weights/Llama-2-7b-chat.gguf\",\n n_gpu_layers=-1,\n n_ctx=4096,\n )\n\n def call_generate(prompt, temperature, max_tokens, stop):\n out = model + prompt + gen(\n name=\"result\",\n max_tokens=max_tokens,\n temperature=temperature,\n stop=stop,\n )\n return out[\"result\"]\n\n else:\n raise ValueError(f\"Invalid backend: {args.backend}\")\n\n def get_one_answer(arg):\n answer = call_generate(**arg, temperature=0)\n states.append(answer)\n\n tic = time.time()\n # we always sequentially execute agent calls to maintain its dependency\n for arg in tqdm(arguments):\n get_one_answer(arg)\n latency = time.time() - tic\n\n print(f\"Latency: {latency:.3f}\")\n\n # Write results\n dump_state_text(f\"tmp_output_{args.backend}.txt\", states)\n\n with open(args.result_file, \"a\") as fout:\n value = {\n \"task\": \"Generative Agents\",\n \"backend\": args.backend,\n \"num_gpus\": 1,\n \"latency\": round(latency, 3),\n # to pack weighted functions as a single agent \n \"num_requests\": len(arguments) / len(mapping),\n \"other\": {\n \"parallel\": args.parallel,\n },\n }\n fout.write(json.dumps(value) + \"\\n\")\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--data-path\", type=str, default=\"agent_calls.jsonl\")\n parser.add_argument(\"--num-events\", type=int, default=10)\n args = add_common_other_args_and_parse(parser)\n main(args)\n"
},
{
"path": "eval/rebase/sglang/benchmark/generative_agents/bench_sglang.py",
"content": "import argparse\nimport json\nimport time\n\nimport sglang as sgl\nfrom sglang.test.test_utils import (\n add_common_sglang_args_and_parse,\n select_sglang_backend,\n)\nfrom sglang.utils import read_jsonl, dump_state_text\n\nfrom agent_functions import (\n poignancy_event,\n generate_event_triple,\n generate_pronunciatio,\n action_location_sector,\n action_location_object,\n)\n\n\ndef main(args):\n lines = read_jsonl(args.data_path)[:args.num_events]\n mapping = {\n \"poignancy_event\": poignancy_event,\n \"generate_event_triple\": generate_event_triple,\n \"generate_pronunciatio\": generate_pronunciatio,\n \"action_location_sector\": action_location_sector,\n \"action_location_object\": action_location_object,\n }\n arguments = [{mapping[k]: v for k, v in l.items()} for l in lines]\n\n # Select backend\n backend = select_sglang_backend(args)\n sgl.set_default_backend(backend)\n\n states = []\n # Run requests\n tic = time.time()\n for a in arguments:\n # only a single key in the dict\n for func, arg in a.items():\n result = func.run(**arg)\n result.sync()\n states.append(result)\n latency = time.time() - tic\n\n # Compute accuracy\n print(f\"Latency: {latency:.3f}\")\n\n # Write results\n dump_state_text(f\"tmp_output_{args.backend}.txt\", states)\n\n with open(args.result_file, \"a\") as fout:\n value = {\n \"task\": \"Generative Agents\",\n \"backend\": args.backend,\n \"num_gpus\": 1,\n \"latency\": round(latency, 3),\n # to pack weighted functions as a single agent\n \"num_requests\": len(arguments) / len(mapping),\n \"other\": {\n \"num_events\": args.num_events,\n \"parallel\": args.parallel,\n },\n }\n fout.write(json.dumps(value) + \"\\n\")\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--data-path\", type=str, default=\"agent_calls.jsonl\")\n parser.add_argument(\"--num-events\", type=int, default=10)\n args = add_common_sglang_args_and_parse(parser)\n main(args)\n"
},
{
"path": "eval/rebase/sglang/benchmark/gsm8k/README.md",
"content": "## Download data\n```\nwget https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/test.jsonl\n```\n\n## Run benchmark\n\n### Benchmark sglang\n```\npython -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000\n```\n\n```\npython3 bench_sglang.py --num-questions 200\n```\n\n\n### Benchmark vllm\n```\npython3 -m vllm.entrypoints.api_server --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000\n```\n\n```\npython3 bench_other.py --num-questions 200 --backend vllm\n```\n\n\n### Benchmark lightllm\n```\n# A10G\npython -m lightllm.server.api_server --tokenizer_mode auto --model_dir ~/model_weights/llama-2-7b-chat-hf --max_total_token_num 16000 --port 22000\n```\n\n```\npython3 bench_other.py --num-questions 200 --backend lightllm\n```\n\n\n### Benchmark guidance\n```\npython3 bench_other.py --num-questions 200 --backend guidance --parallel 1\n```\n\n\n### Benchmark lmql\n```\nCUDA_VISIBLE_DEVICES=0,1 lmql serve-model meta-llama/Llama-2-7b-chat-hf --cuda --port 23000\n```\n\n```\npython3 bench_other.py --num-questions 100 --backend lmql --parallel 2\n```\n"
},
{
"path": "eval/rebase/sglang/benchmark/gsm8k/bench_other.py",
"content": "import argparse\nimport ast\nimport asyncio\nfrom concurrent.futures import ThreadPoolExecutor\nfrom functools import partial\nimport json\nimport re\nimport time\n\nimport numpy as np\nfrom tqdm import tqdm\nfrom sglang.test.test_utils import add_common_other_args_and_parse, call_generate_lightllm, call_generate_vllm, call_generate_srt_raw\nfrom sglang.utils import read_jsonl, dump_state_text\n\n\nINVALID = -9999999\n\n\ndef get_one_example(lines, i, include_answer):\n ret = \"Question: \" + lines[i][\"question\"] + \"\\nAnswer:\" \n if include_answer:\n ret += \" \" + lines[i][\"answer\"]\n return ret\n\n\ndef get_few_shot_examples(lines, k):\n ret = \"\"\n for i in range(k):\n ret += get_one_example(lines, i, True) + \"\\n\\n\"\n return ret\n\n\ndef get_answer_value(answer_str):\n answer_str = answer_str.replace(\",\", \"\")\n numbers = re.findall(r'\\d+', answer_str)\n if len(numbers) < 1:\n return INVALID\n try:\n return ast.literal_eval(numbers[-1])\n except SyntaxError:\n return INVALID\n\n\ndef main(args):\n lines = read_jsonl(args.data_path)\n\n # Construct prompts\n k = args.num_shot\n few_shot_examples = get_few_shot_examples(lines, k)\n\n questions = []\n labels = []\n for i in range(len(lines[:args.num_questions])):\n questions.append(get_one_example(lines, i, False))\n labels.append(get_answer_value(lines[i][\"answer\"]))\n assert all(l != INVALID for l in labels)\n\n states = [None] * len(labels)\n\n # Select backend\n if args.backend == \"lightllm\":\n url = f\"{args.host}:{args.port}/generate\"\n call_generate = partial(call_generate_lightllm, url=url)\n elif args.backend == \"vllm\":\n url = f\"{args.host}:{args.port}/generate\"\n call_generate = partial(call_generate_vllm, url=url)\n elif args.backend == \"srt-raw\":\n url = f\"{args.host}:{args.port}/generate\"\n call_generate = partial(call_generate_srt_raw, url=url)\n elif args.backend == \"guidance\":\n from guidance import models, gen\n\n model = models.LlamaCpp(\"/home/ubuntu/model_weights/Llama-2-7b-chat.gguf\", n_gpu_layers=-1, n_ctx=4096)\n\n def call_generate(prompt, temperature, max_tokens, stop):\n out = model + prompt + gen(name=\"answer\",\n max_tokens=max_tokens, temperature=temperature, stop=stop)\n return out[\"answer\"]\n\n elif args.backend == \"lmql\":\n import lmql\n model = lmql.model(args.model_path,\n endpoint=f\"{args.host}:{args.port}\")\n\n @lmql.query(model=model)\n async def program(question):\n '''lmql\n \"\"\"{question}[ANSWER]\"\"\" where len(TOKENS(ANSWER)) < 257 and STOPS_AT(ANSWER, \"Question\")\n return ANSWER\n '''\n\n async def call_generate(prompt, temperature, max_tokens, stop):\n return await program(question=prompt, temperature=0)\n\n else:\n raise ValueError(f\"Invalid backend: {args.backend}\")\n\n # Run requests\n if args.backend != \"lmql\":\n # Use thread pool\n def get_one_answer(i):\n answer = call_generate(\n prompt=few_shot_examples + questions[i],\n temperature=0,\n max_tokens=256,\n stop=\"Question\")\n states[i] = answer\n\n tic = time.time()\n if args.parallel == 1:\n for i in tqdm(range(len(questions))):\n get_one_answer(i)\n else:\n with ThreadPoolExecutor(args.parallel) as executor:\n executor.map(get_one_answer, list(range(len(questions))))\n else:\n # Use asyncio\n async def batched_call(batch_size):\n for i in range(0, len(questions), batch_size):\n tasks = []\n for q in questions[i:i+batch_size]:\n tasks.append(call_generate(few_shot_examples + q,\n temperature=0, max_tokens=256, stop=\"Question\"))\n rets = await asyncio.gather(*tasks)\n for j in range(len(rets)):\n states[i+j] = rets[j]\n\n tic = time.time()\n asyncio.run(batched_call(batch_size=args.parallel))\n latency = time.time() - tic\n\n preds = []\n for i in range(len(states)):\n preds.append(get_answer_value(states[i]))\n\n # Compute accuracy\n acc = np.mean(np.array(preds) == np.array(labels))\n invalid = np.mean(np.array(preds) == INVALID)\n print(f\"Latency: {latency:.3f}\")\n print(f\"Invalid: {invalid:.3f}\")\n print(f\"Accuracy: {acc:.3f}\")\n\n # Write results\n dump_state_text(f\"tmp_output_{args.backend}.txt\", states)\n\n with open(args.result_file, \"a\") as fout:\n value = {\n \"task\": \"gsm8k\",\n \"backend\": args.backend,\n \"num_gpus\": 1,\n \"latency\": round(latency, 3),\n \"accuracy\": round(acc, 3),\n \"num_requests\": args.num_questions,\n \"other\": {\n \"num_questions\": args.num_questions,\n \"parallel\": args.parallel,\n }\n }\n fout.write(json.dumps(value) + \"\\n\")\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--num-shot\", type=int, default=5)\n parser.add_argument(\"--data-path\", type=str, default=\"test.jsonl\")\n parser.add_argument(\"--num-questions\", type=int, default=200)\n args = add_common_other_args_and_parse(parser)\n main(args)\n"
},
{
"path": "eval/rebase/sglang/benchmark/gsm8k/bench_sglang.py",
"content": "import argparse\nimport ast\nimport json\nimport re\nimport time\n\nimport numpy as np\nfrom sglang.test.test_utils import add_common_sglang_args_and_parse, select_sglang_backend\nfrom sglang.utils import read_jsonl, dump_state_text\n\n\nINVALID = -9999999\n\n\ndef get_one_example(lines, i, include_answer):\n ret = \"Question: \" + lines[i][\"question\"] + \"\\nAnswer:\" \n if include_answer:\n ret += \" \" + lines[i][\"answer\"]\n return ret\n\n\ndef get_few_shot_examples(lines, k):\n ret = \"\"\n for i in range(k):\n ret += get_one_example(lines, i, True) + \"\\n\\n\"\n return ret\n\n\ndef get_answer_value(answer_str):\n answer_str = answer_str.replace(\",\", \"\")\n numbers = re.findall(r'\\d+', answer_str)\n if len(numbers) < 1:\n return INVALID\n try:\n return ast.literal_eval(numbers[-1])\n except SyntaxError:\n return INVALID\n\n\ndef main(args):\n lines = read_jsonl(args.data_path)\n\n # Construct prompts\n k = args.num_shot\n few_shot_examples = get_few_shot_examples(lines, k)\n\n questions = []\n labels = []\n for i in range(len(lines[:args.num_questions])):\n questions.append(get_one_example(lines, i, False))\n labels.append(get_answer_value(lines[i][\"answer\"]))\n assert all(l != INVALID for l in labels)\n arguments = [{\"question\": q} for q in questions]\n\n #####################################\n ######### SGL Program Begin #########\n #####################################\n\n import sglang as sgl\n\n @sgl.function\n def few_shot_gsm8k(s, question):\n s += few_shot_examples + question\n s += sgl.gen(\"answer\", max_tokens=256, stop=\"Question\")\n\n #####################################\n ########## SGL Program End ##########\n #####################################\n\n # Select backend\n backend = select_sglang_backend(args)\n\n # Run requests\n tic = time.time()\n states = few_shot_gsm8k.run_batch(\n arguments, temperature=0, backend=backend, num_threads=args.parallel)\n latency = time.time() - tic\n\n preds = []\n for i in range(len(states)):\n preds.append(get_answer_value(states[i][\"answer\"]))\n\n # Compute accuracy\n acc = np.mean(np.array(preds) == np.array(labels))\n invalid = np.mean(np.array(preds) == INVALID)\n print(f\"Latency: {latency:.3f}\")\n print(f\"Invalid: {invalid:.3f}\")\n print(f\"Accuracy: {acc:.3f}\")\n\n # Write results\n dump_state_text(f\"tmp_output_{args.backend}.txt\", states)\n\n with open(args.result_file, \"a\") as fout:\n value = {\n \"task\": \"gsm8k\",\n \"backend\": args.backend,\n \"num_gpus\": 1,\n \"latency\": round(latency, 3),\n \"accuracy\": round(acc, 3),\n \"num_requests\": args.num_questions,\n \"other\": {\n \"num_questions\": args.num_questions,\n \"parallel\": args.parallel,\n }\n }\n fout.write(json.dumps(value) + \"\\n\")\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--num-shot\", type=int, default=5)\n parser.add_argument(\"--data-path\", type=str, default=\"test.jsonl\")\n parser.add_argument(\"--num-questions\", type=int, default=200)\n args = add_common_sglang_args_and_parse(parser)\n main(args)\n"
},
{
"path": "eval/rebase/sglang/benchmark/hellaswag/README.md",
"content": "## Download data\n```\nwget https://raw.githubusercontent.com/rowanz/hellaswag/master/data/hellaswag_val.jsonl\n```\n\n## Run benchmark\n\n### Benchmark sglang\n```\npython -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000\n```\n\n```\npython3 bench_sglang.py --num-questions 200\n```\n\n\n### Benchmark vllm\n```\npython3 -m vllm.entrypoints.api_server --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000\n```\n\n```\npython3 bench_other.py --num-questions 200 --backend vllm\n```\n\n\n### Benchmark lightllm\n```\n# A10G\npython -m lightllm.server.api_server --tokenizer_mode auto --model_dir ~/model_weights/llama-2-7b-chat-hf --max_total_token_num 16000 --port 22000\n```\n\n```\npython3 bench_other.py --num-questions 200 --backend lightllm\n```\n\n\n### Benchmark guidance\n```\nCUDA_VISIBLE_DEVICES=0,1 python3 bench_other.py --num-questions 200 --backend guidance --parallel 1\n```\n\n\n### Benchmark lmql\n```\nlmql serve-model meta-llama/Llama-2-7b-chat-hf --cuda --port 23000\n```\n\n```\npython3 bench_other.py --num-questions 200 --backend lmql --port 23000 --parallel 1\n```\n"
},
{
"path": "eval/rebase/sglang/benchmark/hellaswag/bench_other.py",
"content": "import argparse\nimport asyncio\nfrom concurrent.futures import ThreadPoolExecutor\nimport json\nfrom functools import partial\nimport time\n\nimport numpy as np\nfrom sglang.test.test_utils import add_common_other_args_and_parse, call_select_lightllm, call_select_vllm\nfrom sglang.utils import read_jsonl\n\n\ndef get_one_example(lines, i, include_answer):\n ret = lines[i][\"activity_label\"] + \": \" + lines[i][\"ctx\"] + \" \"\n if include_answer:\n ret += lines[i][\"endings\"][lines[i][\"label\"]]\n return ret\n\n\ndef get_few_shot_examples(lines, k):\n ret = \"\"\n for i in range(k):\n ret += get_one_example(lines, i, True) + \"\\n\\n\"\n return ret\n\n\ndef main(args):\n lines = read_jsonl(args.data_path)\n\n # Construct prompts\n k = args.num_shot\n few_shot_examples = get_few_shot_examples(lines, k)\n\n questions = []\n choices = []\n labels = []\n for i in range(len(lines[:args.num_questions])):\n questions.append(get_one_example(lines, i, False))\n choices.append(lines[i][\"endings\"])\n labels.append(lines[i][\"label\"])\n\n preds = [None] * len(labels)\n\n # Select backend\n if args.backend == \"lightllm\":\n url = f\"{args.host}:{args.port}/generate\"\n call_select = partial(call_select_lightllm, url=url)\n elif args.backend == \"vllm\":\n url = f\"{args.host}:{args.port}/generate\"\n call_select = partial(call_select_vllm, url=url)\n elif args.backend == \"guidance\":\n from guidance import models, select\n\n model = models.LlamaCpp(\"/home/ubuntu/model_weights/Llama-2-7b-chat.gguf\", n_gpu_layers=-1, n_ctx=4096)\n\n def call_select(context, choices):\n out = model + context + select(choices, name=\"answer\")\n return choices.index(out[\"answer\"])\n\n call_select(\"Hello,\", [\"world\", \"earth\"])\n\n elif args.backend == \"lmql\":\n import lmql\n model = lmql.model(\"meta-llama/Llama-2-7b-chat-hf\",\n endpoint=f\"{args.host}:{args.port}\")\n\n @lmql.query(model=model)\n async def program(ctx, choices):\n '''lmql\n \"\"\"{ctx}[ANSWER]\"\"\" where ANSWER in set(choices)\n return ANSWER\n '''\n\n async def call_select(context, choices):\n answer = await program(ctx=context, choices=choices, temperature=0)\n return choices.index(answer)\n\n else:\n raise ValueError(f\"Invalid backend: {args.backend}\")\n\n # Run requests\n if args.backend != \"lmql\":\n # Use thread pool\n def get_one_answer(i):\n preds[i] = call_select(\n context=few_shot_examples + questions[i],\n choices=choices[i])\n\n tic = time.time()\n if args.parallel == 1:\n for i in range(len(questions)):\n get_one_answer(i)\n else:\n with ThreadPoolExecutor(args.parallel) as executor:\n executor.map(get_one_answer, list(range(len(questions))))\n else:\n # Use asyncio\n async def batched_call(batch_size):\n for i in range(0, len(questions), batch_size):\n tasks = []\n for q, c in zip(questions[i:i+batch_size], choices[i:i+batch_size]):\n tasks.append(call_select(\n context=few_shot_examples + q,\n choices=c))\n rets = await asyncio.gather(*tasks)\n for j in range(len(rets)):\n preds[i+j] = rets[j]\n\n tic = time.time()\n asyncio.run(batched_call(batch_size=args.parallel))\n\n latency = time.time() - tic\n\n # Compute accuracy\n acc = np.mean(np.array(preds) == np.array(labels))\n print(f\"Latency: {latency:.3f}\")\n print(f\"Accuracy: {acc:.3f}\")\n\n # Write results\n with open(args.result_file, \"a\") as fout:\n value = {\n \"task\": \"hellaswag\",\n \"backend\": args.backend,\n \"num_gpus\": 1,\n \"latency\": round(latency, 3),\n \"accuracy\": round(acc, 3),\n \"num_requests\": args.num_questions,\n \"other\": {\n \"num_questions\": args.num_questions,\n \"parallel\": args.parallel,\n }\n }\n fout.write(json.dumps(value) + \"\\n\")\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--num-shot\", type=int, default=20)\n parser.add_argument(\"--data-path\", type=str, default=\"hellaswag_val.jsonl\")\n parser.add_argument(\"--num-questions\", type=int, default=200)\n args = add_common_other_args_and_parse(parser)\n main(args)\n"
},
{
"path": "eval/rebase/sglang/benchmark/hellaswag/bench_sglang.py",
"content": "import argparse\nimport json\nimport time\n\nimport numpy as np\nfrom sglang.test.test_utils import add_common_sglang_args_and_parse, select_sglang_backend\nfrom sglang.utils import read_jsonl\n\n\ndef get_one_example(lines, i, include_answer):\n ret = lines[i][\"activity_label\"] + \": \" + lines[i][\"ctx\"] + \" \"\n if include_answer:\n ret += lines[i][\"endings\"][lines[i][\"label\"]]\n return ret\n\n\ndef get_few_shot_examples(lines, k):\n ret = \"\"\n for i in range(k):\n ret += get_one_example(lines, i, True) + \"\\n\\n\"\n return ret\n\n\ndef main(args):\n lines = read_jsonl(args.data_path)\n\n # Construct prompts\n k = args.num_shot\n few_shot_examples = get_few_shot_examples(lines, k)\n\n questions = []\n choices = []\n labels = []\n for i in range(len(lines[:args.num_questions])):\n questions.append(get_one_example(lines, i, False))\n choices.append(lines[i][\"endings\"])\n labels.append(lines[i][\"label\"])\n arguments = [\n {\"question\": q, \"choices\": c}\n for q, c in zip(questions, choices)\n ]\n\n #####################################\n ######### SGL Program Begin #########\n #####################################\n\n import sglang as sgl\n \n @sgl.function\n def few_shot_hellaswag(s, question, choices):\n s += few_shot_examples + question\n s += sgl.select(\"answer\", choices=choices)\n\n #####################################\n ########## SGL Program End ##########\n #####################################\n\n # Select backend\n backend = select_sglang_backend(args)\n\n # Run requests\n tic = time.time()\n rets = few_shot_hellaswag.run_batch(\n arguments, temperature=0, backend=backend, num_threads=args.parallel)\n preds = [choices[i].index(rets[i][\"answer\"]) for i in range(len(rets))]\n latency = time.time() - tic\n\n # Compute accuracy\n acc = np.mean(np.array(preds) == np.array(labels))\n print(f\"Latency: {latency:.3f}\")\n print(f\"Accuracy: {acc:.3f}\")\n\n # Write results\n with open(args.result_file, \"a\") as fout:\n value = {\n \"task\": \"hellaswag\",\n \"backend\": args.backend,\n \"num_gpus\": 1,\n \"latency\": round(latency, 3),\n \"accuracy\": round(acc, 3),\n \"num_requests\": args.num_questions,\n \"other\": {\n \"num_questions\": args.num_questions,\n \"parallel\": args.parallel,\n }\n }\n fout.write(json.dumps(value) + \"\\n\")\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--num-shot\", type=int, default=20)\n parser.add_argument(\"--data-path\", type=str, default=\"hellaswag_val.jsonl\")\n parser.add_argument(\"--num-questions\", type=int, default=200)\n args = add_common_sglang_args_and_parse(parser)\n main(args)\n"
},
{
"path": "eval/rebase/sglang/benchmark/json_decode_regex/README.md",
"content": "## Run benchmark\n\n### Build dataset\n```\npip install wikipedia\npython3 build_dataset.py\n```\n\n### Dependencies\n\n```\nllama_cpp_python 0.2.19\nguidance 0.1.10\nvllm 0.2.5\noutlines 0.0.22\n```\n\n### Benchmark sglang\n\nRun Llama-7B\n\n```\npython3 -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 \n```\n\nRun Mixtral-8x7B\n\n```\npython3 -m sglang.launch_server --model-path mistralai/Mixtral-8x7B-Instruct-v0.1 --port 30000 --tp-size 8\n```\n\nBenchmark\n\n```\npython3 bench_sglang.py --num-questions 10\n```\n\n\n### Benchmark vllm\n\nRun Llama-7B\n\n```\npython3 -m outlines.serve.serve --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000\n```\n\nBenchmark\n\n```\npython3 bench_other.py --backend vllm --num-questions 10\n```\n\n\n### Benchmark guidance\n\nRun Llama-7B and benchmark\n\n```\npython3 bench_other.py --backend guidance --num-questions 10 --parallel 1\n```\n"
},
{
"path": "eval/rebase/sglang/benchmark/json_decode_regex/bench_other.py",
"content": "import argparse\nimport json\nimport time\nfrom concurrent.futures import ThreadPoolExecutor\nfrom functools import partial\n\nfrom sglang.test.test_utils import (\n add_common_other_args_and_parse,\n call_generate_outlines,\n)\nfrom sglang.utils import dump_state_text, read_jsonl\nfrom sglang.lang.ir import REGEX_INT, REGEX_STRING, REGEX_FLOAT\nfrom tqdm import tqdm\n\nREGEX_LIST = r\"\\[(\" + REGEX_STRING + \", )*\" + REGEX_STRING + r\"\\]\"\n\n\n# fmt: off\ndef json_decode(document, generate):\n s = \"Please extract the information of a city from the following wikipedia page.\\n\"\n s += \"Page begin.\\n\" + document + \"Page end.\\n\"\n s += \"Here is the name, country, and symbol of the city in JSON format.\\n\"\n s += \"{\\n\"\n s += ' \"name\": '\n s += generate(s, max_tokens=8, regex=REGEX_STRING + \",\") + \"\\n\"\n s += ' \"country\": '\n s += generate(s, max_tokens=8, regex=REGEX_STRING + \",\") + \"\\n\"\n s += ' \"latitude\": '\n s += generate(s, max_tokens=8, regex=REGEX_FLOAT + \",\") + \"\\n\"\n s += ' \"population\": '\n s += generate(s, max_tokens=8, regex=REGEX_INT + \",\") + \"\\n\"\n s += ' \"top 3 landmarks\": '\n s += generate(s, max_tokens=24, regex=REGEX_LIST) + \"\\n\"\n s += \"}\\n\"\n\n return s\n# fmt: on\n\n\ndef main(args):\n lines = read_jsonl(args.data_path)\n arguments = []\n for i in range(len(lines[: args.num_questions])):\n arguments.append(\n {\n \"document\": lines[i][\"document\"],\n }\n )\n states = [None] * len(arguments)\n\n # Select backend\n if args.backend == \"vllm\":\n url = f\"{args.host}:{args.port}/generate\"\n generate = partial(call_generate_outlines, url=url, temperature=0)\n elif args.backend == \"guidance\":\n from guidance import gen, models\n\n model = models.LlamaCpp(\n \"/home/ubuntu/model_weights/Llama-2-7b-chat-hf/ggml-model-f16.gguf\",\n n_gpu_layers=-1,\n n_ctx=4096,\n )\n\n def generate(prompt, max_tokens, stop=None, regex=None):\n out = (\n model\n + prompt\n + gen(\n name=\"answer\",\n max_tokens=max_tokens,\n temperature=0,\n stop=stop,\n regex=regex,\n )\n )\n return out[\"answer\"]\n\n # warmup\n for _ in range(3):\n generate(\"Hello!\" * 10, max_tokens=64, stop=None)\n else:\n raise ValueError(f\"Invalid backend: {args.backend}\")\n\n # Run requests\n def get_one_answer(i):\n states[i] = json_decode(generate=generate, **arguments[i])\n\n tic = time.time()\n if args.parallel == 1:\n for i in tqdm(range(len(arguments))):\n get_one_answer(i)\n else:\n with ThreadPoolExecutor(args.parallel) as executor:\n rets = executor.map(get_one_answer, list(range(len(arguments))))\n for _ in rets:\n pass\n\n latency = time.time() - tic\n\n # Compute accuracy\n print(f\"Latency: {latency:.3f}\")\n\n # Write results\n dump_state_text(f\"tmp_output_{args.backend}.txt\", states)\n\n with open(args.result_file, \"a\") as fout:\n value = {\n \"task\": \"json_decode_regex\",\n \"backend\": args.backend,\n \"num_gpus\": 1,\n \"latency\": round(latency, 3),\n \"num_requests\": args.num_questions,\n \"other\": {\n \"parallel\": args.parallel,\n },\n }\n fout.write(json.dumps(value) + \"\\n\")\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--data-path\", type=str, default=\"questions.jsonl\")\n parser.add_argument(\"--num-questions\", type=int, default=20)\n args = add_common_other_args_and_parse(parser)\n main(args)\n"
},
{
"path": "eval/rebase/sglang/benchmark/json_decode_regex/bench_sglang.py",
"content": "import argparse\nimport json\nimport time\n\nimport sglang as sgl\nfrom sglang.lang.ir import REGEX_INT, REGEX_STRING, REGEX_FLOAT\nfrom sglang.test.test_utils import (\n add_common_sglang_args_and_parse,\n select_sglang_backend,\n)\nfrom sglang.utils import dump_state_text, read_jsonl\n\nREGEX_LIST = r\"\\[(\" + REGEX_STRING + \", )*\" + REGEX_STRING + r\"\\]\"\n\n# fmt: off\n@sgl.function\ndef json_warm_up(s):\n s += \"The information about Hogwarts is in the following JSON format.\\n\"\n with s.var_scope(\"json_output\"):\n s += \"{\\n\"\n s += ' \"name\": ' + sgl.gen(\"name\", max_tokens=8, regex=REGEX_STRING + \",\") + \"\\n\"\n s += ' \"country\": ' + sgl.gen(\"country\", max_tokens=8, regex=REGEX_STRING + \",\") + \"\\n\"\n s += ' \"latitude\": ' + sgl.gen(\"latitude\", max_tokens=8, regex=REGEX_FLOAT + \",\") + \"\\n\"\n s += ' \"population\": ' + sgl.gen(\"population\", max_tokens=8, regex=REGEX_INT + \",\") + \"\\n\"\n s += ' \"top 3 landmarks\": ' + sgl.gen( \"landmarks\", max_tokens=24, regex=REGEX_LIST) + \"\\n\"\n s += \"}\\n\"\n print(f'The warmp up json result is:\\n{s[\"json_output\"]}')\n# fmt: on\n\n# fmt: off\n@sgl.function\ndef json_decode(s, document):\n s += \"Please extract the information of a city from the following wikipedia page.\\n\"\n s += \"Page begin.\\n\" + document + \"Page end.\\n\"\n s += \"Here is the name, country, and symbol of the city in JSON format.\\n\"\n with s.var_scope(\"json_output\"):\n s += \"{\\n\"\n s += ' \"name\": ' + sgl.gen(\"name\", max_tokens=8, regex=REGEX_STRING + \",\") + \"\\n\"\n s += ' \"country\": ' + sgl.gen(\"country\", max_tokens=8, regex=REGEX_STRING + \",\") + \"\\n\"\n s += ' \"latitude\": ' + sgl.gen(\"latitude\", max_tokens=8, regex=REGEX_FLOAT + \",\") + \"\\n\"\n s += ' \"population\": ' + sgl.gen(\"population\", max_tokens=8, regex=REGEX_INT + \",\") + \"\\n\"\n s += ' \"top 3 landmarks\": ' + sgl.gen( \"landmarks\", max_tokens=24, regex=REGEX_LIST) + \"\\n\"\n s += \"}\\n\"\n# fmt: on\n\n\ndef main(args):\n lines = read_jsonl(args.data_path)\n arguments = []\n for i in range(len(lines[: args.num_questions])):\n arguments.append(\n {\n \"document\": lines[i][\"document\"],\n }\n )\n\n # Select backend\n backend = select_sglang_backend(args)\n sgl.set_default_backend(backend)\n\n # Warm up\n json_warm_up.run().sync()\n\n # Run requests\n tic = time.time()\n states = json_decode.run_batch(arguments, temperature=0, num_threads=args.parallel)\n latency = time.time() - tic\n\n # Compute accuracy\n print(f\"Latency: {latency:.3f}\")\n\n # Write results\n dump_state_text(f\"tmp_output_{args.backend}.txt\", states)\n\n with open(f\"tmp_{args.backend}_json_results.txt\", \"w\") as fout:\n for state in states:\n fout.write(state[\"json_output\"] + \"\\n\")\n\n with open(args.result_file, \"a\") as fout:\n value = {\n \"task\": \"json_decode_regex\",\n \"backend\": args.backend,\n \"num_gpus\": 1,\n \"latency\": round(latency, 3),\n \"num_requests\": args.num_questions,\n \"other\": {\n \"parallel\": args.parallel,\n },\n }\n fout.write(json.dumps(value) + \"\\n\")\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--data-path\", type=str, default=\"questions.jsonl\")\n parser.add_argument(\"--num-questions\", type=int, default=20)\n args = add_common_sglang_args_and_parse(parser)\n main(args)\n"
},
{
"path": "eval/rebase/sglang/benchmark/json_decode_regex/build_dataset.py",
"content": "import json\n\nimport transformers\nimport wikipedia\n\nmodel_path = \"meta-llama/Llama-2-7b-chat-hf\"\nt = transformers.AutoTokenizer.from_pretrained(model_path)\ncity_names = [\n \"los angles\",\n \"london\",\n \"tokyo\",\n \"beijing\",\n \"singapore\",\n \"paris\",\n \"dubai\",\n \"sydney\",\n \"moscow\",\n \"rome\",\n \"toronto\",\n \"rio de janeiro\",\n \"istanbul\",\n \"berlin\",\n \"auckland\",\n \"buenos aires\",\n \"mexico city\",\n \"mumbai\",\n \"seoul\",\n \"bangkok\",\n \"cairo\",\n \"athens\",\n \"jerusalem\",\n]\n\n\ndef get_content(city_name):\n content = str(wikipedia.page(city_name).content)\n content = content.replace(\"\\n\\n\", \"\\n\")\n\n tokens = t.encode(content)\n\n expected_tokens = 3000\n truncate_len = int((expected_tokens / len(tokens)) * len(content))\n truncate_content = content[:truncate_len]\n truncate_tokens = t.encode(truncate_content)\n\n # Count token\n print(\n f\"city_name: {city_name}, #tokens: {len(tokens)}, #truncate tokens: {len(truncate_tokens)}\"\n )\n\n return truncate_content\n\n\nif __name__ == \"__main__\":\n with open(\"questions.jsonl\", \"w\") as fout:\n for city_name in city_names:\n truncate_content = get_content(city_name)\n fout.write(json.dumps({\"document\": truncate_content}) + \"\\n\")\n"
},
{
"path": "eval/rebase/sglang/benchmark/json_jump_forward/README.md",
"content": "## Run benchmark\n\n### Dependencies\n\n```\nllama_cpp_python 0.2.38\nguidance 0.1.10\nvllm 0.2.7\noutlines 0.0.25\n```\n\n### Build dataset\n\nWhen benchmarking long document information retrieval, run the following command to build the dataset:\n\n```bash\npip install wikipedia\npython3 build_dataset.py\n```\n\n### Benchmark sglang\n\nRun Llama-7B\n\n```bash\npython3 -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 \n```\n\nBenchmark Character Generation\n\n```bash\npython3 bench_sglang.py --mode character\n```\n\nBenchmark City Information Retrieval\n\n```bash\npython3 bench_sglang.py --mode city\n```\n\n\n### Benchmark vllm\n\nRun Llama-7B\n\n```bash\npython3 -m outlines.serve.serve --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000\n```\n\nBenchmark Character Generation\n\n```bash\npython3 bench_other.py --mode character --backend vllm\n```\n\nBenchmark City Information Retrieval\n\n```bash\npython3 bench_other.py --mode city --backend vllm\n```\n\n### Benchmark guidance\n\nRun Llama-7B and benchmark character generation\n\n```bash\npython3 bench_other.py --mode character --backend guidance --parallel 1\n```\n\nRun Llama-7B and benchmark city information retrieval\n\n```bash\npython3 bench_other.py --mode city --backend guidance --parallel 1\n```\n"
},
{
"path": "eval/rebase/sglang/benchmark/json_jump_forward/bench_other.py",
"content": "import argparse\nimport json\nimport time\nfrom concurrent.futures import ThreadPoolExecutor\nfrom functools import partial\n\nimport guidance\nfrom sglang.test.test_utils import (\n add_common_other_args_and_parse,\n call_generate_outlines,\n)\nfrom sglang.utils import dump_state_text, read_jsonl\nfrom tqdm import tqdm\n\n# there are some FSM bugs with json regex converted from pydantic model\n# here use a string regex instead\n# regex_string = build_regex_from_object(HarryPoterRole)\ncharacter_regex = (\n r\"\"\"\\{\\n\"\"\"\n + r\"\"\" \"name\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"house\": \"(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)\",\\n\"\"\"\n + r\"\"\" \"blood status\": \"(Pure-blood|Half-blood|Muggle-born)\",\\n\"\"\"\n + r\"\"\" \"occupation\": \"(student|teacher|auror|ministry of magic|death eater|order of the phoenix)\",\\n\"\"\"\n + r\"\"\" \"wand\": \\{\\n\"\"\"\n + r\"\"\" \"wood\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"core\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"length\": [0-9]{1,2}\\.[0-9]{0,2}\\n\"\"\"\n + r\"\"\" \\},\\n\"\"\"\n + r\"\"\" \"alive\": \"(Alive|Deceased)\",\\n\"\"\"\n + r\"\"\" \"patronus\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"bogart\": \"[\\w\\d\\s]{1,16}\"\\n\"\"\"\n + r\"\"\"\\}\"\"\"\n)\n\ncity_regex = (\n r\"\"\"\\{\\n\"\"\"\n + r\"\"\" \"name\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"country\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"latitude\": [-+]?[0-9]*\\.?[0-9]{0,2},\\n\"\"\"\n + r\"\"\" \"population\": [-+]?[0-9]{1,9},\\n\"\"\"\n + r\"\"\" \"top 3 landmarks\": \\[\"[\\w\\d\\s]{1,16}\", \"[\\w\\d\\s]{1,16}\", \"[\\w\\d\\s]{1,16}\"\\]\\n\"\"\"\n + r\"\"\"\\}\"\"\"\n)\n\n# fmt: off\ndef character_gen(name, generate):\n s = name + \" is a character in Harry Potter. Please fill in the following information about this character.\\n\"\n s += generate(s, max_tokens=256, regex=character_regex)\n return s\n# fmt: on\n\n# fmt: off\ndef city_gen(document, generate):\n s = \"Please extract the information of a city from the following wikipedia page.\\n\"\n s += \"Page begin.\\n\" + document + \"Page end.\\n\"\n s += \"Here is the name, country, and symbol of the city in JSON format.\\n\"\n s += generate(s, max_tokens=256, regex=city_regex)\n return s\n# fmt: on\n\n\n@guidance\ndef character_maker(lm, name):\n regex_str_no_quote = r\"[\\w\\d\\s]+\"\n regex_float = r\"[0-9]+\\.[0-9]+\"\n lm += f\"\"\"\\\n {name} is a character in Harry Potter. Please fill in the following information about this character.\n {{\n \"name\": \"{guidance.gen(\"name\", max_tokens=16, regex=regex_str_no_quote)}\",\n \"house\": \"{guidance.select(options=['Gryffindor', 'Slytherin', 'Ravenclaw', 'Hufflepuff'], name='house')}\",\n \"blood status\": \"{guidance.select(options=['Pure-blood', 'Half-blood', 'Muggle-born'], name='blood status')}\",\n \"occupation\": \"{guidance.select(options=['student', 'teacher', 'auror', 'ministry of magic', 'death eater', 'order of the phoenix'], name='occupation')}\",\n \"wand\": {{\n \"wood\": \"{guidance.gen(\"wood\", max_tokens=16, regex=regex_str_no_quote)}\",\n \"core\": \"{guidance.gen('core', max_tokens=16, regex=regex_str_no_quote)}\",\n \"length\": {guidance.gen('length', max_tokens=10, regex=regex_float)}\n }},\n \"alive\": \"{guidance.select(options=['Alive', 'Deceased'], name='alive')}\",\n \"patronus\": \"{guidance.gen('patronus', max_tokens=16, regex=regex_str_no_quote)}\",\n \"bogart\": \"{guidance.gen('bogart', max_tokens=16, regex=regex_str_no_quote)}\"\n }}\n \"\"\"\n\n return lm\n\n\n@guidance\ndef city_maker(lm, document):\n regex_str_no_quote = r\"[\\w\\d\\s]+\"\n regex_float = r\"[0-9]+\\.[0-9]+\"\n lm += f\"\"\"\\\n Please extract the information of a city from the following wikipedia page.\n Page begin.\n {document}\n Page end.\n Here is the name, country, and symbol of the city in JSON format.\n {{\n \"name\": \"{guidance.gen(\"name\", max_tokens=16, regex=regex_str_no_quote)}\",\n \"country\": \"{guidance.gen(\"country\", max_tokens=16, regex=regex_str_no_quote)}\",\n \"latitude\": {guidance.gen(\"latitude\", max_tokens=10, regex=regex_float)},\n \"population\": {guidance.gen(\"population\", max_tokens=10, regex=r\"[0-9]+\")},\n \"top 3 landmarks\": [\n \"{guidance.gen(\"landmark1\", max_tokens=16, regex=regex_str_no_quote)}\", \"{guidance.gen(\"landmark2\", max_tokens=16, regex=regex_str_no_quote)}\", \"{guidance.gen(\"landmark3\", max_tokens=16, regex=regex_str_no_quote)}\"\n ]\n }}\n \"\"\"\n\n return lm\n\n\ndef bench_character(args):\n arguments = []\n with open(args.data_path, \"r\") as f:\n for line in f:\n arguments.append({\"name\": line.strip()})\n arguments = arguments[: args.num_jsons]\n\n states = [None] * len(arguments)\n\n # Select backend\n if args.backend == \"vllm\":\n url = f\"{args.host}:{args.port}/generate\"\n generate = partial(call_generate_outlines, url=url, temperature=0)\n\n def func(i):\n states[i] = character_gen(**arguments[i], generate=generate)\n\n get_one_answer = func\n elif args.backend == \"guidance\":\n model = guidance.models.LlamaCpp(\n args.llama_cpp_model_path,\n n_gpu_layers=-1,\n n_ctx=4096,\n )\n\n def func(i):\n lm = model + character_maker(**arguments[i])\n states[i] = lm\n\n get_one_answer = func\n else:\n raise ValueError(f\"Invalid backend: {args.backend}\")\n\n tic = time.time()\n if args.parallel == 1:\n for i in tqdm(range(len(arguments))):\n get_one_answer(i)\n else:\n with ThreadPoolExecutor(args.parallel) as executor:\n rets = executor.map(get_one_answer, list(range(len(arguments))))\n for _ in rets:\n pass\n\n latency = time.time() - tic\n\n return states, latency\n\n\ndef bench_city_doc(args):\n arguments = []\n for line in read_jsonl(args.data_path):\n arguments.append({\"document\": line[\"document\"]})\n arguments = arguments[: args.num_jsons]\n\n states = [None] * len(arguments)\n\n # Select backend\n if args.backend == \"vllm\":\n url = f\"{args.host}:{args.port}/generate\"\n generate = partial(call_generate_outlines, url=url, temperature=0)\n\n def func(i):\n states[i] = city_gen(**arguments[i], generate=generate)\n\n get_one_answer = func\n elif args.backend == \"guidance\":\n model = guidance.models.LlamaCpp(\n args.llama_cpp_model_path,\n n_gpu_layers=-1,\n n_ctx=4096,\n )\n\n def func(i):\n lm = model + city_maker(**arguments[i])\n states[i] = lm\n\n get_one_answer = func\n else:\n raise ValueError(f\"Invalid backend: {args.backend}\")\n\n tic = time.time()\n if args.parallel == 1:\n for i in tqdm(range(len(arguments))):\n get_one_answer(i)\n else:\n with ThreadPoolExecutor(args.parallel) as executor:\n rets = executor.map(get_one_answer, list(range(len(arguments))))\n for _ in rets:\n pass\n\n latency = time.time() - tic\n\n return states, latency\n\n\ndef main(args):\n if args.mode == \"character\":\n args.data_path = \"dataset.txt\"\n states, latency = bench_character(args)\n elif args.mode == \"city\":\n args.data_path = \"questions.jsonl\"\n states, latency = bench_city_doc(args)\n\n # Compute accuracy\n print(f\"Latency: {latency:.3f}\")\n\n # Write results\n dump_state_text(f\"tmp_output_{args.backend}_{args.mode}.txt\", states)\n\n with open(args.result_file, \"a\") as fout:\n value = {\n \"task\": \"json_jump_forward\",\n \"backend\": args.backend,\n \"latency\": round(latency, 3),\n \"num_jsons\": args.num_jsons,\n \"mode\": args.mode,\n \"parallel\": args.parallel,\n }\n fout.write(json.dumps(value) + \"\\n\")\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--data-path\", type=str)\n parser.add_argument(\"--num-jsons\", type=int, default=50)\n parser.add_argument(\n \"--mode\", type=str, default=\"character\", choices=[\"character\", \"city\"]\n )\n parser.add_argument(\n \"--llama-cpp-model-path\",\n type=str,\n default=\"/home/ubuntu/model_weights/Llama-2-7b-chat-hf/ggml-model-f16.gguf\",\n )\n args = add_common_other_args_and_parse(parser)\n main(args)\n"
},
{
"path": "eval/rebase/sglang/benchmark/json_jump_forward/bench_sglang.py",
"content": "import argparse\nimport json\nimport time\n\nimport sglang as sgl\nfrom sglang.test.test_utils import (\n add_common_sglang_args_and_parse,\n select_sglang_backend,\n)\nfrom sglang.utils import dump_state_text, read_jsonl\n\n# there are some FSM bugs with json regex converted from pydantic model\n# here use a string regex instead\n# regex_string = build_regex_from_object(HarryPoterRole)\ncharacter_regex = (\n r\"\"\"\\{\\n\"\"\"\n + r\"\"\" \"name\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"house\": \"(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)\",\\n\"\"\"\n + r\"\"\" \"blood status\": \"(Pure-blood|Half-blood|Muggle-born)\",\\n\"\"\"\n + r\"\"\" \"occupation\": \"(student|teacher|auror|ministry of magic|death eater|order of the phoenix)\",\\n\"\"\"\n + r\"\"\" \"wand\": \\{\\n\"\"\"\n + r\"\"\" \"wood\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"core\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"length\": [0-9]{1,2}\\.[0-9]{0,2}\\n\"\"\"\n + r\"\"\" \\},\\n\"\"\"\n + r\"\"\" \"alive\": \"(Alive|Deceased)\",\\n\"\"\"\n + r\"\"\" \"patronus\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"bogart\": \"[\\w\\d\\s]{1,16}\"\\n\"\"\"\n + r\"\"\"\\}\"\"\"\n)\n\ncity_regex = (\n r\"\"\"\\{\\n\"\"\"\n + r\"\"\" \"name\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"country\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n + r\"\"\" \"latitude\": [-+]?[0-9]*\\.?[0-9]{0,2},\\n\"\"\"\n + r\"\"\" \"population\": [-+]?[0-9]{1,9},\\n\"\"\"\n + r\"\"\" \"top 3 landmarks\": \\[\"[\\w\\d\\s]{1,16}\", \"[\\w\\d\\s]{1,16}\", \"[\\w\\d\\s]{1,16}\"\\]\\n\"\"\"\n + r\"\"\"\\}\"\"\"\n)\n\n# fmt: off\n@sgl.function\ndef character_gen(s, name):\n s += name + \" is a character in Harry Potter. Please fill in the following information about this character.\\n\"\n s += sgl.gen(\"json_output\", max_tokens=256, regex=character_regex)\n# fmt: on\n\n# fmt: off\n@sgl.function\ndef city_gen(s, document):\n s += \"Please extract the information of a city from the following wikipedia page.\\n\"\n s += \"Page begin.\\n\" + document + \"Page end.\\n\"\n s += \"Here is the name, country, and symbol of the city in JSON format.\\n\"\n s += sgl.gen(\"json_output\",max_tokens=256, regex=city_regex)\n# fmt: on\n\n\ndef bench_city_doc(args):\n arguments = []\n for line in read_jsonl(args.data_path):\n arguments.append({\"document\": line[\"document\"]})\n arguments = arguments[: args.num_jsons]\n\n # Select backend\n backend = select_sglang_backend(args)\n sgl.set_default_backend(backend)\n\n # Run requests\n tic = time.time()\n states = city_gen.run_batch(\n arguments,\n temperature=0,\n num_threads=args.parallel,\n progress_bar=(args.parallel == 1),\n )\n latency = time.time() - tic\n\n return states, latency\n\n\ndef bench_character(args):\n arguments = []\n with open(args.data_path, \"r\") as f:\n for line in f:\n arguments.append({\"name\": line.strip()})\n arguments = arguments[: args.num_jsons]\n\n # Select backend\n backend = select_sglang_backend(args)\n sgl.set_default_backend(backend)\n\n # Run requests\n tic = time.time()\n states = character_gen.run_batch(\n arguments,\n temperature=0,\n num_threads=args.parallel,\n progress_bar=(args.parallel == 1),\n )\n latency = time.time() - tic\n\n return states, latency\n\n\ndef main(args):\n if args.mode == \"character\":\n args.data_path = \"dataset.txt\"\n states, latency = bench_character(args)\n elif args.mode == \"city\":\n args.data_path = \"questions.jsonl\"\n states, latency = bench_city_doc(args)\n\n # Compute accuracy\n print(f\"Latency: {latency:.3f}\")\n\n # Write results\n dump_state_text(f\"tmp_output_{args.backend}_{args.mode}.txt\", states)\n with open(f\"{args.backend}_{args.mode}.json\", \"w\") as fout:\n for state in states:\n fout.write(state[\"json_output\"] + \"\\n\")\n\n with open(args.result_file, \"a\") as fout:\n value = {\n \"task\": \"json_jump_forward\",\n \"backend\": args.backend,\n \"latency\": round(latency, 3),\n \"num_jsons\": args.num_jsons,\n \"mode\": args.mode,\n \"parallel\": args.parallel,\n }\n fout.write(json.dumps(value) + \"\\n\")\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--data-path\", type=str)\n parser.add_argument(\"--num-jsons\", type=int, default=50)\n parser.add_argument(\n \"--mode\", type=str, default=\"character\", choices=[\"character\", \"city\"]\n )\n args = add_common_sglang_args_and_parse(parser)\n main(args)\n"
},
{
"path": "eval/rebase/sglang/benchmark/json_jump_forward/build_dataset.py",
"content": "import json\n\nimport transformers\nimport wikipedia\n\nmodel_path = \"meta-llama/Llama-2-7b-chat-hf\"\nt = transformers.AutoTokenizer.from_pretrained(model_path)\ncity_names = [\n \"los angles\",\n \"london\",\n \"tokyo\",\n \"beijing\",\n \"singapore\",\n \"paris\",\n \"dubai\",\n \"sydney\",\n \"moscow\",\n \"rome\",\n \"toronto\",\n \"rio de janeiro\",\n \"istanbul\",\n \"berlin\",\n \"auckland\",\n \"buenos aires\",\n \"mexico city\",\n \"mumbai\",\n \"seoul\",\n \"bangkok\",\n \"cairo\",\n \"athens\",\n \"jerusalem\",\n]\n\n\ndef get_content(city_name):\n content = str(wikipedia.page(city_name).content)\n content = content.replace(\"\\n\\n\", \"\\n\")\n\n tokens = t.encode(content)\n\n expected_tokens = 3000\n truncate_len = int((expected_tokens / len(tokens)) * len(content))\n truncate_content = content[:truncate_len]\n truncate_tokens = t.encode(truncate_content)\n\n # Count token\n print(\n f\"city_name: {city_name}, #tokens: {len(tokens)}, #truncate tokens: {len(truncate_tokens)}\"\n )\n\n return truncate_content\n\n\nif __name__ == \"__main__\":\n with open(\"questions.jsonl\", \"w\") as fout:\n for city_name in city_names:\n truncate_content = get_content(city_name)\n fout.write(json.dumps({\"document\": truncate_content}) + \"\\n\")\n"
},
{
"path": "eval/rebase/sglang/benchmark/json_jump_forward/dataset.txt",
"content": "Harry Potter\nHermione Granger\nRon Weasley\nAlbus Dumbledore\nSeverus Snape\nRubeus Hagrid\nDraco Malfoy\nGinny Weasley\nFred Weasley\nGeorge Weasley\nPercy Weasley\nSirius Black\nRemus Lupin\nNeville Longbottom\nLuna Lovegood\nCedric Diggory\nCho Chang\nLord Voldemort\nMinerva McGonagall\nFilius Flitwick\nDolores Umbridge\nBellatrix Lestrange\nLucius Malfoy\nMolly Weasley\nArthur Weasley\nNymphadora Tonks\nDobby\nMoaning Myrtle\nPeter Pettigrew\nAlastor 'Mad-Eye' Moody\nHorace Slughorn\nVernon Dursley\nPetunia Dursley\nDudley Dursley\nArgus Filch\nSybill Trelawney\nGilderoy Lockhart\nFleur Delacour\nViktor Krum\nBill Weasley\nOliver Wood\nCornelius Fudge\nBarty Crouch Sr.\nBarty Crouch Jr.\nKingsley Shacklebolt\nQuirinus Quirrell\nNearly Headless Nick\nAunt Marge\nGriphook\nLudo Bagman"
},
{
"path": "eval/rebase/sglang/benchmark/latency_throughput/README.md",
"content": "### Download data\n```\nwget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json\n```\n\n### SGLang\n```\npython -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000\n```\n\n```\npython3 bench_throughput.py --backend srt --tokenizer meta-llama/Llama-2-7b-chat-hf --dataset ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 10 --request-rate 10 --port 30000\n```\n\n\n### vLLM\n```\npython3 -m vllm.entrypoints.api_server --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --swap-space 16 --port 21000\n```\n\n```\npython3 bench_throughput.py --backend vllm --tokenizer meta-llama/Llama-2-7b-chat-hf --dataset ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 10 --request-rate 10 --port 21000\n```\n\n\n### LightLLM\n```\npython -m lightllm.server.api_server --model_dir ~/model_weights/Llama-2-7b-chat-hf --max_total_token_num 15600 --tokenizer_mode auto --port 22000\n```\n\n```\npython3 bench_throughput.py --backend lightllm --tokenizer meta-llama/Llama-2-7b-chat-hf --dataset ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 10 --request-rate 10 --port 22000\n```\n"
},
{
"path": "eval/rebase/sglang/benchmark/latency_throughput/bench_throughput.py",
"content": "\"\"\"Benchmark online serving throughput.\n\nOn the server side, run one of the following commands:\n (vLLM backend)\n python -m vllm.entrypoints.api_server \\\n --model \n