What is the difference between a starry day and a solar day. And how come a day actually lasts exactly 23h 56min?

Answer

A solar day is the average time it takes the earth to perform one rotation in relation to the sun. That takes 24 hours. Why ? Because we defined the hour that way. That 24 hours is therefore the average time that the sun needs between two successive passages due south.

A sidereal day is the time it takes the earth to perform one rotation opposite the fixed starry sky, and that takes 23h56m. How come ?

Look at the attached figure. On the left is the Earth, on two consecutive days. You also see the sun, and there is a star infinitely distant to the right. At a certain point on the first day you see the sun and the star in the same direction (suppose this is possible), due south.
During the next day the earth starts to move in its orbit (to the 2nd position) and meanwhile rotates around its own axis, counterclockwise.
After 11:56 p.m., the Earth has again turned in the direction of that distant star, but because it has moved in its orbit in the meantime, the sun is now just to the left of that star. The earth has to rotate for a while to get the sun directly in the south again. That takes four minutes extra, and after 24 hours the sun is back in the south. At that moment the star is slightly to the right of the sun, in those 24 hours that star has traveled slightly more than 360° in the sky.

So the star comes to the south on average every 23h56 minutes, the average every 24h. The reason is that the earth itself also moves around the sun.