# Confident AI

Aiolum story: https://aiolum.com/story/confident-ai/ · added 2026-11-12 · updated 2026-11-12

**In short.** Confident AI (Cloud platform for evaluating and monitoring LLM applications, built around the open-source DeepEval package, United States): 19.5K GitHub stars on DeepEval, November 2026. Evidence grade E: no revenue figure was checked. Team: 2 co-founders; 7 people on the Y Combinator profile read in November 2026. Launched August 2023; still running.

Cloud platform for evaluating and monitoring LLM applications, built around the open-source DeepEval package · United States · AI-native · team of 2 to 10 · still running · group: Developer tools and coding agents

- Outcome: 19.5K GitHub stars on DeepEval, November 2026 (evidence grade of the headline number: E; no revenue figure)
- Launch: August 2023; months from launch to the outcome: 39
- Team: 2 co-founders; 7 people on the Y Combinator profile read in November 2026
- Outside funding: Seed-stage per the company's careers page; Y Combinator Winter 2025; amount not found
- Main factor: Distribution
- Meters, 1 to 4: evidence strength 1, copyable pattern 2, capital needed 2
- Tags: AI-native, Small team, Funded, Running, B2B, Developer tools

Links:

- Visit confident-ai.com: https://www.confident-ai.com/

**The product.** A hosted platform that tests and monitors applications built on large language models. Engineers write tests with the open-source DeepEval package. The platform stores the datasets, shows the results to the whole team and watches production traffic.

**Why it worked.** The founders gave away the testing package first and sold the cloud platform to the teams that already used it. The package was created on GitHub in August 2023. The cloud platform has a free plan and self-serve paid plans of US$200 and US$2,000 a month, as listed by the company.

Evidence grades: A = disclosed by a public company, a filing or a payment processor, or verified by one; B = two or more independent reports agree, or a journalist confirmed it; C = a founder claim that no one else verified; D = reported, not verified: one founder claim or a third-party estimate with no second source; E = no revenue figure found, so the headline is a non-money fact and the other facts carry their own grades; calc = computed on this page from graded facts.

## Summary

Confident AI is a cloud platform for LLM evaluation, the practice of scoring the answers of a language model against test cases. Engineers write the tests in DeepEval, an open-source Python package. The platform adds shared datasets, test reports, production tracing and alerts.

Two founders started it: Jeffrey Ip as CEO and Kritin Vongthongsri as CTO, per the Y Combinator profile. The DeepEval repository on GitHub was created on 10 August 2023, when the site carried the earlier name Twilix Inc. On 20 February 2025 the founders posted a launch thread on Hacker News and wrote that the platform "runs on a freemium tier".

No revenue figure was found, so this story has grade E. The headline is the number of GitHub stars on DeepEval, which counts users of the free package, not paying customers. The pricing page lists self-serve plans, but no source states that a customer has paid.

## Why did Confident AI work?

### It released a free package first and built the paid platform around it

The Launch HN post says the founders spent a year building DeepEval before adding the platform. The careers page says the company started by building DeepEval. The free code has an Apache 2.0 license. The platform adds a dataset editor, shared reports and tracing for teams.

### It writes its tests into the engineers' own code pipelines

The founders wrote that DeepEval runs in the test pipelines of enterprises. They describe it as pytest for language models. Pytest is a common Python test tool. A test that runs on every code change is a habit that a paid team plan can build on.

### It lists three self-serve prices and one custom price

The pricing page shows a free plan, a US$200 plan and a US$2,000 plan, all with a button to try for free. Only Enterprise needs a talk with sales. The page also sells trace storage at US$1 per GB-month. That puts a price on usage next to the seat plans.

## How it was built

These details come from the company's own pages and the founders' launch post. They are grade C.

- The evaluation metrics use a language model as the judge. The founders wrote that this method can be inconsistent.
- The sub-processors page names OpenAI for model processing, Supabase and ClickHouse for data, and AWS for blob storage.
- The same page names Stripe for payments, Mixpanel and PostHog for analytics, and Pylon for customer support.
- The home page says the platform has SDKs in Python and TypeScript and works with tracing by OpenTelemetry.

Sources: [11, 5, 1]

## Who tried this before Confident AI, and why did they fail?

Three companies in our database of failed startups sold developer tools for testing or understanding software. The values below are from that database and are not independently verified.

The failed startups come from our database of dead startups, retold here in our own words; Its entries are AI-assisted summaries of public sources, so they carry no evidence grade.

### Testmunk (2012 to 2015; funding US$120K burned; cause: Crowded market)

- Promise: A mobile app testing platform that runs automated tests on real devices to cut time to market.
- Why it died: The record says it could not stand apart in a quickly saturating market of automated testing, with larger rivals named.
- Record: 

### CodeSee (2020 to 2024; funding US$10.0M burned; cause: Value not proven)

- Promise: Automatic visual maps of code, so developers can understand and change large codebases faster.
- Why it died: The record says it built a vitamin when the market demanded painkillers, and that it struggled to turn free users into paying customers.
- Record: 

### Kite (2014 to 2024; funding US$21.0M burned; cause: Too early, weak pricing)

- Promise: An AI code completion engine in the editor that suggested code for developers.
- Why it died: The record calls it too early for a market that later grew, with missteps in how it earned money.
- Record: 

## What changed

All three records sold a tool that developers could replace or do without. The Testmunk record names larger testing platforms, the CodeSee record names GitHub and the Kite record names GitHub Copilot. The CodeSee and Kite records point to trouble in earning money from free users.

Confident AI also gives its core package away for free. Its pricing page adds paid cloud plans for teams, but no source here says how many teams pay. The risk that remains is the same conversion from free use to paid use.

## Can one person copy Confident AI now?

**Verdict: Partly. The open-source start can be copied; the credibility of the metrics takes work.**

A small team can release a free developer package and sell a hosted team version around it. The founders wrote that evaluation quality depends on the quality of datasets and metrics. Others can follow the structure, but trust in the metrics comes from years of use.

1. Pick one task that engineers already do by hand, such as testing model outputs.
2. Release the core as open-source code that runs in their existing test pipeline.
3. Sell the shared parts: stored datasets, reports, alerts and team access.
4. Publish a free plan and one self-serve price, and keep a custom plan for large companies.

**What is still open.** Other jobs that engineers run in code and that teams need to share, such as safety tests, cost tracking or prompt versioning.

## How much revenue does Confident AI make? The numbers, graded

No revenue figure is public. The facts below show a free package, a price list, a small funded team and company statements about customers

| Fact | Value | Grade | Note | Sources |
|---|---|---|---|---|
| Open-source package created | 10 August 2023 | C | The GitHub programming interface gives this creation date for the repository confident-ai/deepeval. The repository description reads "The LLM Evaluation Framework". | [14] |
| Company start | 2023, under the earlier name Twilix Inc. | C | The company was formerly named Twilix Inc. and was founded by August 2023. An archived copy of confident-ai.com from 15 August 2023 reads "Copyright @ 2023 Twilix Inc.". The Y Combinator profile reads "Founded: 2024". No company register was read. | [17, 12] |
| Launch of the cloud platform thread | 20 February 2025 | C | The founders posted "Launch HN" on that date. They wrote that DeepEval had been built over the past year and that the platform "runs on a freemium tier". | [11] |
| Founders | Jeffrey Ip (CEO), Kritin Vongthongsri (CTO) | C | The Y Combinator profile names both as founders. The Hacker News post is signed by the user jeffreyip, who writes "we're Jeffrey and Kritin". | [12, 11] |
| Team size | 7 people | C | Y Combinator lists "Team Size: 7" on a profile read in November 2026. The profile has no date. The careers page says the company is "a small team". | [12, 6] |
| Funding | seed-stage; amount not found | C | The careers page says "We're seed-stage". No page read states a funding amount or an investor. | [6, 12] |
| Plans and prices | Free, Starter US$200 a month, Team US$2,000 a month, Enterprise custom | C | The company's pricing page, read on 12 November 2026. The Free plan has a "Try Now For Free" button. The FAQ says an upgrade is charged by prorated amount. | [2] |
| How customers pay | credit cards for self-serve plans; Stripe | C | The pricing FAQ says all major credit cards are accepted for self-serve plans. The privacy policy names Stripe as a payment processor. | [2, 4] |
| GitHub stars on DeepEval | 19.5K stars, 2,194 forks | C | The repository page shows 19.5k stars on 12 November 2026. The programming interface gives 19,451 stars. The package is free and licensed under Apache 2.0. | [13, 14] |
| Named customers | Finom, RLDatix, Humach, Amdocs, Supernormal | C | The case studies page lists four of them, and RLDatix has a case study page of its own. Each case study is the company's own text with a named customer quotation. No source states what they pay. | [7, 8, 9, 10] |
| Usage claims | 600K evaluations a day (founders, February 2025) | C | The founders wrote in the Launch HN post that DeepEval ran over 600K evaluations daily in the test pipelines of named enterprises. Later pages give larger numbers; see the watch section. | [11, 12] |
| Time from the package to the headline | 39 months | calc | Calculated: August 2023 to November 2026. | [14, 13] |

How to read this table. Grade C is a statement by the company, its founders or a platform page. A grade E story has no dated money figure, so no row here is a revenue number.

## Pricing

Confident AI prices as read on the pricing page on 12 November 2026. The page shows one monthly price per plan and no yearly price

| Plan | Per month, billed yearly (US$) | Per month, billed monthly (US$) | Limit |
|---|---|---|---|
| Free | 0 | 0 | 2 user seats, 1 project, 5 test runs a week, 1 GB-month of trace spans |
| Starter |  | 200 | unlimited user seats, 5 projects, 5 GB-months of trace spans |
| Team |  | 2000 | unlimited user seats and projects, 75 GB-months of trace spans |
| Enterprise |  |  | custom pricing; contact sales; on-premises deployment and 24x7 support |

Starter and Team are billed per organization per month, per the pricing page. The page says self-serve plans accept all major credit cards. Trace storage beyond the included amount costs US$1 per GB-month.

## When did Confident AI launch, and how did it grow? Timeline

From an open-source package in 2023 to a cloud platform with a price list in 2026. Each entry carries its grade.

- **10 August 2023** (grade C [14, 17]): The DeepEval repository is created on GitHub. Five days later an archived copy of confident-ai.com shows the company name Twilix Inc.
- **Spring 2024** (grade D [11]): A commenter on the 2025 launch thread writes that they tried the company's tests and test dashboard in the spring of the year before.
- **2024** (grade C [12]): Y Combinator lists the company as founded in 2024, in its Winter 2025 batch, in San Francisco.
- **20 February 2025** (grade C [11]): The founders post Launch HN: a cloud platform for DeepEval with a freemium tier. The post has 117 points.
- **29 October 2026** (grade C [15]): The company blog announces Confident Voice, which checks calls made by voice agents.
- **12 November 2026** (grade C [13, 2]): The DeepEval repository shows 19.5K stars. The company lists a Free plan and Starter and Team plans.

## What could go wrong for Confident AI?

- **What is not verified.** No revenue figure was found. The pages read were the company's site, its terms, privacy and case-study pages, the Y Combinator profile, the launch thread and GitHub. No payment-processor profile, filing, press piece with a figure or funding amount was found. The pricing page shows self-serve plans with prices, billing terms and a card payment answer, and the privacy policy names Stripe. No source states that a customer has paid. The case studies are the company's own text, and no independent source confirms them.
- **Counters, badges and testimonials.** The home page says "TRUSTED BY 500+ LEADING AI COMPANIES" with no list or date, and an "Evals ran to date" counter whose value loads by script. Y Combinator gives 2 million evaluations a day and 3 million monthly downloads in one launch post, and 400K monthly downloads in a founder profile. The founders wrote 600K evaluations a day in February 2025. SOC 2 Type II, HIPAA and GDPR badges have no certificate here. Five named quotations on the home page match the company's case study pages. One more quotation is unnamed: the initials "SD", a "Senior Director of Engineering" at a "Fortune 500 medical device company".
- **Start date and package versus platform.** The GitHub stars count the free package, not paying customers. The repository was created in August 2023, and Y Combinator lists the company as founded in 2024. The earliest archived copy of confident-ai.com, of 15 August 2023, has the title "Twilix - Evaluate your LLMs in Production". Its footer reads "Copyright @ 2023 Twilix Inc.". The terms of 2026 name Confident AI, Inc. No source read says when Twilix Inc. was founded or when the name changed.
- **What the product tests.** The platform tests and watches the customer's own AI applications, including voice agents and security tests of those applications. The same company publishes DeepTeam, an open-source red teaming framework. A company blog article explains jailbreak techniques and tells readers to test their own application. The marketing speaks to engineers, product managers and quality teams. The privacy policy says the services are not intended for children under 13.

## Sources

1. Confident AI home page — https://www.confident-ai.com/ (grade C). Read in full, 12 November 2026. Source for the product description, the counters, the badges and the three named quotations.
2. Confident AI pricing page — https://www.confident-ai.com/pricing (grade C). Read in full, 12 November 2026. Source for the plans, the prices, the limits and the billing FAQ.
3. Confident AI terms of service — https://www.confident-ai.com/terms (grade C). Read in full, 12 November 2026. Last modified 2 March 2026. Source for the fee terms and for the legal name, Confident AI, Inc.
4. Confident AI privacy policy — https://www.confident-ai.com/privacy-policy (grade C). Read in full, 12 November 2026. Last modified 26 May 2026. Source for Stripe and for the children's privacy section.
5. Confident AI list of sub-processors — https://www.confident-ai.com/subprocessors-list (grade C). Read in full, 12 November 2026. Last modified 18 February 2026. Source for the cloud, payment and analytics providers.
6. Confident AI careers page — https://www.confident-ai.com/careers (grade C). Read in full, 12 November 2026. Source for seed-stage, the start with DeepEval and the description of the team.
7. Confident AI case studies page — https://www.confident-ai.com/case-study (grade C). Read in full, 12 November 2026. Source for four of the five named customers; RLDatix is on its own case study page.
8. Confident AI case study: Finom — https://www.confident-ai.com/case-study/finom (grade C). Read in full, 12 November 2026. Company text with a named quotation. Source for the quotation on the home page.
9. Confident AI case study: RLDatix — https://www.confident-ai.com/case-study/rldatix (grade C). Read in full, 12 November 2026. Company text with a named quotation. Source for the quotation on the home page.
10. Confident AI case study: Humach — https://www.confident-ai.com/case-study/humach (grade C). Read in full, 12 November 2026. Company text with named quotations. Source for the quotation on the home page.
11. Launch HN: Confident AI (YC W25), Hacker News, 20 February 2025 — https://news.ycombinator.com/item?id=43116633 (grade C). Read in full, with the comments, 12 November 2026. Founder post. Source for the launch, the freemium tier and the 600K claim.
12. Confident AI at Y Combinator — https://www.ycombinator.com/companies/confident-ai (grade C). Read in full, 12 November 2026. Source for the founders, the start year, the batch and the team size.
13. GitHub repository confident-ai/deepeval — https://github.com/confident-ai/deepeval (grade C). Read the repository page, 12 November 2026. Source for the star count, the license and the link to Confident AI.
14. GitHub programming interface, repository confident-ai/deepeval — https://api.github.com/repos/confident-ai/deepeval (grade C). Read in full, 12 November 2026. Source for the creation date, 19,451 stars and 2,194 forks.
15. Confident AI blog — https://www.confident-ai.com/blog (grade C). Read the first page of the list, 12 November 2026. Source for the 29 October 2026 post on Confident Voice.
16. Database of failed startups, local export — /method/#dead (grade C). Record pages read on the website on 12 November 2026. AI-written summaries, not independently verified. Source for the failed predecessors.
17. Internet Archive copy of confident-ai.com, 15 August 2023 — http://web.archive.org/web/20230815165611/https://www.confident-ai.com/ (grade C). Read in full, 12 November 2026. The earliest stored copy. Source for the earlier name Twilix Inc. and its 2023 copyright line.

Contact: aiolum@livyden.com. Run Confident AI? Send corrections or a verified figure to aiolum@livyden.com (subject: Correction: Confident AI).