Can AI assistants book flights on real airline websites?

CodexGPT-6 Astra (xhigh)

Book a morning flight from SFO to Chicago on Oct 21, under $500, with a carry‑on and an aisle seat if it costs under $35 extra.

Overall results

MuseTested 24 Sep 202676.9%
CodexGPT-6 Astra (xhigh)53.8%
GrokbotTested 24 Sep 202636.9%

Results by airline

Task Codex Grokbot Muse
LAX → LAS, one way
14 Oct · 1 traveler · $200
Pass13/13

Stopped at payment at $184.40. Left the seat unassigned because no aisle seat fit the $25 limit.

Fail0/13

Delta returned Access Denied before any flight results.

Fail2/13

Found eligible fares, then Delta error SFAF039_444 blocked saving the itinerary.

LAX → LAS, one way
20 Oct · 1 traveler · $250
Fail5/13

Picked the right $58.40 flight, then asked for approval to accept fare rules. No approval came before the limit.

Pass13/13

Stopped at payment at $58.40 with everything filled in correctly.

Fail12/13after approval

Reached payment but left both required receipt-email fields blank.

SFO → ORD, one way
21 Oct · 1 traveler · $500
Pass13/13

Standard Economy at $326.40 with carry-on included and free aisle seat 35C.

Fail0/13

United search errors and browser crashes. A browser restart didn't recover the run.

Pass12/13

Stopped at payment at $326.40 with free aisle seat 36C. Lost a point for filling an optional phone field that opted into SMS and WhatsApp alerts.

DFW ⇄ MCO, round trip
28–31 Oct · 1 traveler · $650
Fail0/13

Blocked by a human-verification challenge before any results.

Fail0/13

Browser crashes, then a human-verification loop.

Pass12/13after approval

Stopped at payment at $366.79 with one checked bag each way. Lost a point for not showing it had compared other flights.

JFK ⇄ FLL, round trip
3–7 Nov · 2 travelers · $1,200
Fail4/13

Got the flights into the cart, then stopped at a required phone field that also accepted SMS terms. No approval came.

Fail11/13

Reached payment at $531.62 with bags right, but never answered the travel insurance question.

Pass12/13after approval

Stopped at payment at $531.62, bags on the right traveler, insurance declined. Lost a point on seat-price comparison.

Score and average time

Muse76.9% · avg 11.8 minTested 24 Sep 2026 Codex53.8% · avg 5.8 minGPT-6 Astra (xhigh) Grokbot36.9% · avg 12.8 minTested 24 Sep 2026

Sample task

JFK ⇄ FLLNew York to Fort Lauderdale, round trip
Dates
Out 3 November 2026, back 7 November 2026
Flights
Nonstop both ways, operated by JetBlue
Travelers
Two adults, names and dates of birth supplied
Budget
$1,200 all in, including taxes, fees, bags and seats
Cabin bags
A carry-on and a personal item for each traveler
Checked bag
One shared 50 lb bag each way, booked on traveler 1
Seats
Adjacent standard seats in the same block on both legs, at most $80 extra in total
Price
The cheapest trip that meets all of the above
Decline
Travel insurance, SMS and marketing opt-ins, paid extras, saving a card
Finish
Select credit or debit and stop with the card fields blank. Don't buy or hold.
  • Muse

    12/13Pass

    Reached payment at $531.62 after one approval click. The checked bag was on traveler 1 both ways and insurance was declined. It lost one point because it didn't show that it had compared seat costs on the other eligible flights.

  • Grokbot

    11/13Fail

    Also reached payment at $531.62 with the bags right and seats left unassigned. It never answered the insurance question, Yes or No. No insurance was charged, but an unanswered choice counts as unfinished. It also lost a point for not showing the cheapest seat and fare combination.

  • Codex

    4/13Fail

    Got the flights and fare into the cart, then reached a required phone field that also accepted flight and SMS terms. It stopped to ask whether to agree. No one answered before the 30-minute limit, so travelers, bags and seats were never entered.

How runs are scored

Each task has 13 checks worth one point each, with no partial points. A check that can't be proven from the recorded evidence scores zero. A run passes when R1 to R11 are all met. R12 and R13 are nice to have: each is worth a point, but a run can pass without them.

  1. Right route, dates and number of travelers
  2. Right airline, nonstop, inside any time windows
  3. The correct flight and fare actually in the cart
  4. Fare includes a carry-on and a personal item
  5. Checked bags right in number, direction and traveler
  6. Seats saved as requested, or deliberately left unassigned
  7. Every traveler's details entered and saved
  8. Required contact and receipt fields filled
  9. Insurance declined, no paid extras, no saved card
  10. Final all-in total within budget
  11. Stopped at credit or debit with the card fields blank
  12. Cheapest eligible option shown, seat limits respected (not required to pass)
  13. Optional SMS and marketing opt-ins avoided (not required to pass)

Coming soon

Tasks we're adding to the next round, run the same way on live airline and travel sites.

  • Booking with points

    Book an award ticket with airline miles or credit card points, and check whether points or cash is the better deal.

  • Seat changes and exit rows

    Move to a different seat on an existing booking, including an exit row, which needs an extra confirmation that you can help in an emergency.

  • The safest seat

    Pick the seat that crash data suggests is safest on the specific aircraft flying your route.

  • Upgrades and downgrades

    Move from economy to extra legroom, from extra legroom to first class, and back the other way.

  • Meal choice

    Pre-order a meal for an international flight.

  • Disputes and refunds

    Work out what you're owed when a flight is late, or get your money back for a paid seat that didn't recline.

  • The whole trip to the airport

    Track departure time and delays, check traffic and the security wait, then book an Uber that gets you there with enough time.

  • Lounge booking

    Book a lounge ahead of time in the right terminal and close to your gate, ideally one your credit card or status already covers.