Aiolum Browse stories
All stories
Cloud platform for evaluating and monitoring LLM applications, built around the open-source DeepEval package · United States

Confident AI: Product, team and how it was built

Small teamFundedB2BDeveloper tools

Confident AI (Cloud platform for evaluating and monitoring LLM applications, built around the open-source DeepEval package, United States): 19.5K GitHub stars on DeepEval, November 2026. Evidence grade E: no revenue figure was checked. Team: 2 co-founders; 7 people on the Y Combinator profile read in November 2026. Launched August 2023; still running.Added

Outcome
19.5K GitHub stars on DeepEval, November 2026
Evidence
E no revenue figure was checked
Launch
August 2023
Status
Still running
Team
2 co-founders; 7 people on the Y Combinator profile read in November 2026
Outside funding
Seed-stage per the company's careers page; Y Combinator Winter 2025; amount not found
Main channel
Open source
No revenue figure

What is not verified

No revenue figure was found. The pages read were the company's site, its terms, privacy and case-study pages, the Y Combinator profile, the launch thread and GitHub. No payment-processor profile, filing, press piece with a figure or funding amount was found. The pricing page shows self-serve plans with prices, billing terms and a card payment answer, and the privacy policy names Stripe. No source states that a customer has paid. The case studies are the company's own text, and no independent source confirms them. This story is grade E: the business is real and its product, price and customers are described from public pages, but no revenue figure was checked. Aiolum shows it as an idea worth studying, not as proof that it makes money.

Confident AI is a cloud platform for LLM evaluation, the practice of scoring the answers of a language model against test cases. Engineers write the tests in DeepEval, an open-source Python package. The platform adds shared datasets, test reports, production tracing and alerts.

Two founders started it: Jeffrey Ip as CEO and Kritin Vongthongsri as CTO, per the Y Combinator profile. The DeepEval repository on GitHub was created on 10 August 2023, when the site carried the earlier name Twilix Inc. On 20 February 2025 the founders posted a launch thread on Hacker News and wrote that the platform "runs on a freemium tier".

No revenue figure was found, so this story has grade E. The headline is the number of GitHub stars on DeepEval, which counts users of the free package, not paying customers. The pricing page lists self-serve plans, but no source states that a customer has paid.

Confident AI: 19.5K GitHub stars on DeepEval, November 2026. Cloud platform for evaluating and monitoring LLM applications, built around the open-source DeepEval package, United States, team of 2 to 10, evidence grade E.
The story in one card. Share it with the address of this page.
01

Why did Confident AI work?

Four causes, in the order they mattered. Each one is a reusable lesson.

01

It released a free package first and built the paid platform around it

The Launch HN post says the founders spent a year building DeepEval before adding the platform. The careers page says the company started by building DeepEval. The free code has an Apache 2.0 license. The platform adds a dataset editor, shared reports and tracing for teams.

02

It writes its tests into the engineers' own code pipelines

The founders wrote that DeepEval runs in the test pipelines of enterprises. They describe it as pytest for language models. Pytest is a common Python test tool. A test that runs on every code change is a habit that a paid team plan can build on.

03

It lists three self-serve prices and one custom price

The pricing page shows a free plan, a US$200 plan and a US$2,000 plan, all with a button to try for free. Only Enterprise needs a talk with sales. The page also sells trace storage at US$1 per GB-month. That puts a price on usage next to the seat plans.

How it was built

These details come from the company's own pages and the founders' launch post. They are grade C. [11, 5, 1]

  • The evaluation metrics use a language model as the judge. The founders wrote that this method can be inconsistent.
  • The sub-processors page names OpenAI for model processing, Supabase and ClickHouse for data, and AWS for blob storage.
  • The same page names Stripe for payments, Mixpanel and PostHog for analytics, and Pylon for customer support.
  • The home page says the platform has SDKs in Python and TypeScript and works with tracing by OpenTelemetry.

Tools named in the sources

Each one opens its page in the Tools tab, with every story that uses it.

02

Who tried this before Confident AI, and why did they fail?

Three companies in our database of failed startups sold developer tools for testing or understanding software. The values below are from that database and are not independently verified.

Died 2012 to 2015Burned US$120K burned

Testmunk

The promise

A mobile app testing platform that runs automated tests on real devices to cut time to market.

Why it died

The record says it could not stand apart in a quickly saturating market of automated testing, with larger rivals named.

Crowded market
Died 2020 to 2024Burned US$10.0M burned

CodeSee

The promise

Automatic visual maps of code, so developers can understand and change large codebases faster.

Why it died

The record says it built a vitamin when the market demanded painkillers, and that it struggled to turn free users into paying customers.

Value not proven
Died 2014 to 2024Burned US$21.0M burned

Kite

The promise

An AI code completion engine in the editor that suggested code for developers.

Why it died

The record calls it too early for a market that later grew, with missteps in how it earned money.

Too early, weak pricing

The failed startups come from our database of 1,749 dead startups, retold here in our own words. Its entries are AI-assisted summaries of public sources, so these cards carry no evidence grade.

What changed

All three records sold a tool that developers could replace or do without. The Testmunk record names larger testing platforms, the CodeSee record names GitHub and the Kite record names GitHub Copilot. The CodeSee and Kite records point to trouble in earning money from free users.

Confident AI also gives its core package away for free. Its pricing page adds paid cloud plans for teams, but no source here says how many teams pay. The risk that remains is the same conversion from free use to paid use.

03

Can one person copy Confident AI now?

Partly. The open-source start can be copied; the credibility of the metrics takes work.

A small team can release a free developer package and sell a hosted team version around it. The founders wrote that evaluation quality depends on the quality of datasets and metrics. Others can follow the structure, but trust in the metrics comes from years of use.

Where it is still open: Other jobs that engineers run in code and that teams need to share, such as safety tests, cost tracking or prompt versioning.

  1. 1Pick one task that engineers already do by hand, such as testing model outputs.
  2. 2Release the core as open-source code that runs in their existing test pipeline.
  3. 3Sell the shared parts: stored datasets, reports, alerts and team access.
  4. 4Publish a free plan and one self-serve price, and keep a custom plan for large companies.
04

How much revenue does Confident AI make?

Each row shows the evidence grade and the source numbers from the list at the end of this page. The grades are explained in How Aiolum verifies.

No revenue figure is public. The facts below show a free package, a price list, a small funded team and company statements about customers
FactValueGradeHow we knowSources
Open-source package created10 August 2023CThe GitHub programming interface gives this creation date for the repository confident-ai/deepeval. The repository description reads "The LLM Evaluation Framework".[14]
Company start2023, under the earlier name Twilix Inc.CThe company was formerly named Twilix Inc. and was founded by August 2023. An archived copy of confident-ai.com from 15 August 2023 reads "Copyright @ 2023 Twilix Inc.". The Y Combinator profile reads "Founded: 2024". No company register was read.[17, 12]
Launch of the cloud platform thread20 February 2025CThe founders posted "Launch HN" on that date. They wrote that DeepEval had been built over the past year and that the platform "runs on a freemium tier".[11]
FoundersJeffrey Ip (CEO), Kritin Vongthongsri (CTO)CThe Y Combinator profile names both as founders. The Hacker News post is signed by the user jeffreyip, who writes "we're Jeffrey and Kritin".[12, 11]
Team size7 peopleCY Combinator lists "Team Size: 7" on a profile read in November 2026. The profile has no date. The careers page says the company is "a small team".[12, 6]
Fundingseed-stage; amount not foundCThe careers page says "We're seed-stage". No page read states a funding amount or an investor.[6, 12]
Plans and pricesFree, Starter US$200 a month, Team US$2,000 a month, Enterprise customCThe company's pricing page, read on 12 November 2026. The Free plan has a "Try Now For Free" button. The FAQ says an upgrade is charged by prorated amount.[2]
How customers paycredit cards for self-serve plans; StripeCThe pricing FAQ says all major credit cards are accepted for self-serve plans. The privacy policy names Stripe as a payment processor.[2, 4]
GitHub stars on DeepEval19.5K stars, 2,194 forksCThe repository page shows 19.5k stars on 12 November 2026. The programming interface gives 19,451 stars. The package is free and licensed under Apache 2.0.[13, 14]
Named customersFinom, RLDatix, Humach, Amdocs, SupernormalCThe case studies page lists four of them, and RLDatix has a case study page of its own. Each case study is the company's own text with a named customer quotation. No source states what they pay.[7, 8, 9, 10]
Usage claims600K evaluations a day (founders, February 2025)CThe founders wrote in the Launch HN post that DeepEval ran over 600K evaluations daily in the test pipelines of named enterprises. Later pages give larger numbers; see the watch section.[11, 12]
Time from the package to the headline39 monthscalcCalculated: August 2023 to November 2026.[14, 13]

How to read this table. Grade C is a statement by the company, its founders or a platform page. A grade E story has no dated money figure, so no row here is a revenue number.

Confident AI prices as read on the pricing page on 12 November 2026. The page shows one monthly price per plan and no yearly price
PlanPer month, billed yearly (US$)Per month, billed monthly (US$)Limit
Free002 user seats, 1 project, 5 test runs a week, 1 GB-month of trace spans
Starter200unlimited user seats, 5 projects, 5 GB-months of trace spans
Team2000unlimited user seats and projects, 75 GB-months of trace spans
Enterprisecustom pricing; contact sales; on-premises deployment and 24x7 support

Starter and Team are billed per organization per month, per the pricing page. The page says self-serve plans accept all major credit cards. Trace storage beyond the included amount costs US$1 per GB-month.

05

When did Confident AI launch, and how did it grow?

From an open-source package in 2023 to a cloud platform with a price list in 2026. Each entry carries its grade.

  1. 10 August 2023CThe DeepEval repository is created on GitHub. Five days later an archived copy of confident-ai.com shows the company name Twilix Inc. [14, 17]
  2. Spring 2024DA commenter on the 2025 launch thread writes that they tried the company's tests and test dashboard in the spring of the year before. [11]
  3. 2024CY Combinator lists the company as founded in 2024, in its Winter 2025 batch, in San Francisco. [12]
  4. 20 February 2025CThe founders post Launch HN: a cloud platform for DeepEval with a freemium tier. The post has 117 points. [11]
  5. 29 October 2026CThe company blog announces Confident Voice, which checks calls made by voice agents. [15]
  6. 12 November 2026CThe DeepEval repository shows 19.5K stars. The company lists a Free plan and Starter and Team plans. [13, 2]

What could go wrong for Confident AI?

The risks are the mirror image of the causes above.

What is not verified

No revenue figure was found. The pages read were the company's site, its terms, privacy and case-study pages, the Y Combinator profile, the launch thread and GitHub. No payment-processor profile, filing, press piece with a figure or funding amount was found. The pricing page shows self-serve plans with prices, billing terms and a card payment answer, and the privacy policy names Stripe. No source states that a customer has paid. The case studies are the company's own text, and no independent source confirms them.

Counters, badges and testimonials

The home page says "TRUSTED BY 500+ LEADING AI COMPANIES" with no list or date, and an "Evals ran to date" counter whose value loads by script. Y Combinator gives 2 million evaluations a day and 3 million monthly downloads in one launch post, and 400K monthly downloads in a founder profile. The founders wrote 600K evaluations a day in February 2025. SOC 2 Type II, HIPAA and GDPR badges have no certificate here. Five named quotations on the home page match the company's case study pages. One more quotation is unnamed: the initials "SD", a "Senior Director of Engineering" at a "Fortune 500 medical device company".

Start date and package versus platform

The GitHub stars count the free package, not paying customers. The repository was created in August 2023, and Y Combinator lists the company as founded in 2024. The earliest archived copy of confident-ai.com, of 15 August 2023, has the title "Twilix - Evaluate your LLMs in Production". Its footer reads "Copyright @ 2023 Twilix Inc.". The terms of 2026 name Confident AI, Inc. No source read says when Twilix Inc. was founded or when the name changed.

What the product tests

The platform tests and watches the customer's own AI applications, including voice agents and security tests of those applications. The same company publishes DeepTeam, an open-source red teaming framework. A company blog article explains jailbreak techniques and tells readers to test their own application. The marketing speaks to engineers, product managers and quality teams. The privacy policy says the services are not intended for children under 13.

Sources

The grade next to each source is the best grade that the source can support. "Read in full" means that the text of the page was downloaded and the numbers were checked in it.

  1. C Confident AI home page
    Read in full, 12 November 2026. Source for the product description, the counters, the badges and the three named quotations.
  2. C Confident AI pricing page
    Read in full, 12 November 2026. Source for the plans, the prices, the limits and the billing FAQ.
  3. C Confident AI terms of service
    Read in full, 12 November 2026. Last modified 2 March 2026. Source for the fee terms and for the legal name, Confident AI, Inc.
  4. C Confident AI privacy policy
    Read in full, 12 November 2026. Last modified 26 May 2026. Source for Stripe and for the children's privacy section.
  5. C Confident AI list of sub-processors
    Read in full, 12 November 2026. Last modified 18 February 2026. Source for the cloud, payment and analytics providers.
  6. C Confident AI careers page
    Read in full, 12 November 2026. Source for seed-stage, the start with DeepEval and the description of the team.
  7. C Confident AI case studies page
    Read in full, 12 November 2026. Source for four of the five named customers; RLDatix is on its own case study page.
  8. C Confident AI case study: Finom
    Read in full, 12 November 2026. Company text with a named quotation. Source for the quotation on the home page.
  9. C Confident AI case study: RLDatix
    Read in full, 12 November 2026. Company text with a named quotation. Source for the quotation on the home page.
  10. C Confident AI case study: Humach
    Read in full, 12 November 2026. Company text with named quotations. Source for the quotation on the home page.
  11. C Launch HN: Confident AI (YC W25), Hacker News, 20 February 2025
    Read in full, with the comments, 12 November 2026. Founder post. Source for the launch, the freemium tier and the 600K claim.
  12. C Confident AI at Y Combinator
    Read in full, 12 November 2026. Source for the founders, the start year, the batch and the team size.
  13. C GitHub repository confident-ai/deepeval
    Read the repository page, 12 November 2026. Source for the star count, the license and the link to Confident AI.
  14. C GitHub programming interface, repository confident-ai/deepeval
    Read in full, 12 November 2026. Source for the creation date, 19,451 stars and 2,194 forks.
  15. C Confident AI blog
    Read the first page of the list, 12 November 2026. Source for the 29 October 2026 post on Confident Voice.
  16. C Database of failed startups, local export
    Record pages read on the website on 12 November 2026. AI-written summaries, not independently verified. Source for the failed predecessors.
  17. C Internet Archive copy of confident-ai.com, 15 August 2023
    Read in full, 12 November 2026. The earliest stored copy. Source for the earlier name Twilix Inc. and its 2023 copyright line.

Run this company? Send corrections or a verified figure to aiolum@livyden.com

Back to all stories