I'm guessing you already have the 'hard facts' - the ones on the side of the package. Instead, I'll give you my own experience of the benefits of quitting.
I smoked for several years in much the same as you. Three to six cigarettes a day, usually at times of the day when I wanted to get started or have a moment of reflection. I loved coffee, cigarettes, and conversation in the morning. I loved "cocktail hour" with cigarettes and conversation in the evening. My smoking wasn't so much a physical addiction as a movement addiction - something to do with my hands. So why quit?

• I got tired of smelling like cigarettes. When I caught a cold, the smell really hit me afterward until my nose got used to the smell again. Then it hit me - I bet non-smokers think I smell like this pretty much all the time! And non-smokers are now 75% - 80% of the American Population, depending on your source. (http://tinyurl.com/2nazaq)
  
• I started getting into good foods, and smoking interfered with my taste. We started making our own beer and chinese food to begin with, and have continued to become serious cooks. The taste of good food and drink are much better after quitting. 
  
• I started martial arts, which required better fitness level and breathing. Although I wasn't addicted to smoking, I could feel the effect it had on my lungs. This got better relatively quickly, by the way.
  
• Then there was the smoke break at work. In one of my jobs, smokers have a real edge by taking a break every 90 minutes or so at the break table outside. That edge consists of informal meetings where the politics of the organization and backstory are shared. I decided not to give this up. I continued to go out when I needed a break, as it was discriminatory to me NOT to allow non-smokers this advantage. I was questioned on this once, when a supervisor observed "You don't smoke!" I replied "I can start..." and reached to borrow a cigarette. She said "NO!" and that was the end of it.

So, in summary, I can breathe deeply and smell almost unusually well (maybe I didn't know what I was missing before). Taste dramatically improved, probably a little more slowly than breathing. My house, car, and clothing don't reek of cigarettes. I still have the benefits of the break, and not the stigma of being a smoker. Major win!
Oh, and then there was the expense. I applied the cost of cigarettes to other things - in my case, clothes - directly. So every cigarette not smoked was clothing on my back, literally. With the cost of cigarettes today, that would be a big chunk of change!

This question hits me personaly because I've lost people close to my from smoking:

• I lost my grandfather to emphazima when I was fourteen - he was seventy-five.  His life would've been longer and he would've enjoyed it more had he not smoked.  All of my life, I remember him hacking and coughing all the time.
• I lost my favorite uncle in 2005 to smoking - he only had 20% of his lung capacity when he passed away in his 60's.
• We lost a close family friend (he was my fiance's uncle) to lung cancer yesterday - he was seventy-one.  He's another that would've had a longer life had it not been for smoking.

I smoked from age eleven to age twenty-two and quit cold turkey when I decided to have children.  I became a mother two years later and I don't miss smoking at all.  As soon as I learned the truth behind the cause of my Grandfather's death (which was sheltered from me for years), it solidified and added validation to my decision.

I've been trying to help my friends and family quit ever since I smoked my last cigarette in 1997.  If I can do it, anyone can

The risk of lung cancer increases 100% for each cigarette you smoke.  So 3-4 cigarettes a day would mean you are 3-4 times more likely to get lung cancer than a non-smoker.

Is that a risk you're willing to take?

Even if you're willing to accept it for yourself, there are risks to others from both second and Third hand Smoke.

you may read http://www.ways-to-quit-smoking.com/Stop-Smoking/help-quit-smoking-tips-to-quit-smoking and http://www.ways-to-quit-smoking.com/why-is-it-hard-to-quit-smoking.html

Yes, you should quit smoking.  Be sure to watch the video.Here are a bunch of reasons why:

According to someone who has smoked for 25 years, there are a lot of symptoms and side affects. Smokers generally speaking do not live as long as non smokers who have the same health habits other than smoking. You can suffer from lung cancer, emphysema, bad breath, tobacco stained teeth, and more.

These symptoms include:

1. A pressure, heaviness, and often pain in my lungs in the morning.

2. Morning cough.

3. Loss of my singing voice.

4. Breathlessness in my reading voice.

5. Unpleasant irritation of the mucous membranes of my mouth.

6. General tiredness and loss of energy.

7. Headaches.

8. I realize that I need to take naps to combat the tiredness I feel because my body has to process all the poisons that cigs have.

9. Cigarette hangovers after late nights out.

10. The need to exercise early in the day, before having smoked a lot.

11. Waking up at night coughing or with tickles in my throat.

Psychological Reasons to Quit Smoking:

1. Complete self-hate and loss of self-esteem.

2. A constant feeling of guilt at what I am doing to my body and the consequences that my smoking might have on my children, i.e. the care they might need to give me later in life.

Yes, you definitely should.