Is the exercise working?

Ive quit smoking after 15+ yrs and began swimming on a daily basis. I swim after work most days at least 5 times a week. I can swim 500 metres freestyle and thats it. I started off doing 250 metres freestyle and 250 breast stroke but the breast stroke is too easy and I managed to replace it with just freestyle. Ive been doing this for about 10 weeks now and I don't seem to be improving. Im still totally a wreck when Im done and I still need to stop for breath every 50 metres or risk sinking and drowning. I would have thought Id be able to swim further and not have to stop as often after 10 weeks but Ive really only improved modestly since I started. All the while people twice my age are swimming in the next lane who never stop for breath and continue swimming after Ive left the pool. Why is it taking so long for me to get fit? Im 33 male and not overweight. I swim till my lungs cant take anymore and my body and limbs hurt. I swim intensely and often. My aim is to reach a standard of being able to swim 2km freestyle without stopping. When I can do this Ill know Im fit but I dont know if I ever will. Please help.