Previous studies suggested that severe epilepsies, e.g., developmental and epileptic encephalopathies (DEEs), are mainly caused by ultra-rare de novo genetic variants. For milder disease, rare genetic variants could contribute to the phenotype. To determine the importance of rare variants for different epilepsy types, we analyzed a whole-exome sequencing cohort of 9,170 epilepsy-affected individuals and 8,436 control individuals. Here, we separately analyzed three different groups of epilepsies: severe DEEs, genetic generalized epilepsy (GGE), and non-acquired focal epilepsy (NAFE). We required qualifying rare variants (QRVs) to occur in control individuals with an allele count ≥ 1 and a minor allele frequency ≤ 1:1,000, to be predicted as deleterious (CADD ≥ 20), and to have an odds ratio in individuals with epilepsy ≥ 2. We identified genes enriched with QRVs primarily in NAFE (n = 72), followed by GGE (n = 32) and DEE (n = 21). This suggests that rare variants may play a more important role for causality of NAFE than for DEE. Moreover, we found that genes harboring QRVs, e.g., HSGP2, FLNA, or TNC, encode proteins that are involved in structuring the brain extracellular matrix. The present study confirms an involvement of rare variants for NAFE that occur also in the general population, while in DEE and GGE, the contribution of such variants appears more limited.