Ustad Abdul Karim Khan was born on 11 November 1875 in Kirana town in Shamli, Uttar Pradesh in a family with music tradition which traced its roots to musician brothers Ghulam Ali and Ghulam Maula. His father, Kale Khan, was the grandson of Ghulam Ali. Abdul Karim Khan received training from his uncle Abdullah Khan and father Kale Khan. He also received guidance from another uncle, Nanhe Khan. Abdul Karim Khan was musically influenced by Ustad Rahmat Khan of Gwalior Gharana. Apart from vocals and sarangi, he also learned the veena(been), sitar, and tabla.

According to a story, he was initially a sarangi player but decided to switch to vocals because of the low status of sarangi players at that time. In his early years, he used to sing with his brother Abdul Haq. They appeared before the Murtaza Khan of the "Maula Bakhsh Khandan", then all appeared before the Maharaja (king). They were all influenced by Murtaza's style. The king of Baroda State was impressed by this singing duo and made them the court musicians.

This is where he met Tarabai Mane, who was the daughter of Sardar Maruti Rao Mane, a member of the royal family. When they decided to get married, they were ousted from Baroda. The couple settled in Bombay. In 1922, Tarabai Mane left Abdul Karim Khan, which had a major impact on his music, making it pensive and meditative.

The innovations he brought to his vocal style distinguish Kirana style from others. The slow melodic development of the raga in Vilambit laya (slow tempo) was the most characteristic aspect of his music. He worked hard to maintain his voice to be sweet and melodious which shaped his music. The Thumri style he developed is also quite different from the 'Poorab ang' or 'Punjabi ang'. His thumri progresses at a leisurely slow pace with ample abandonment. He was also the first Hindustani musician to seriously study the Carnatic system and probably the first to be invited to sing all over the South. He has even recorded a Thyagaraja Krithi. He was also influenced by Rehmat Khan of the Gawaliar Gharana and adopted the direct style of presentation."The ustad was an expert on many musical instruments, especially the veena and the sarangi. An expert in repairing musical instruments, he carried with him his set of tools for repairs everywhere.

He is the ustad of the legendary Roshan Ara Begum.