Size most definitely matters when it comes to the grips on your handgun. Working in and around the firearms industry for the last 13 years there is a common trend I've seen over and over again: The assumption that certain people who might be smaller in stature need smaller guns and smaller calibers - "The smaller the person, the smaller the gun", right? Wrong. Whether you're 6'8" tall or 4'10" tall there is almost no handgun that would be impossible for you shoot comfortably. Except maybe for the massive S&W 500 revolver or the tiny derringer that fits in a belt buckle... those aren't fun to shoot no matter what size you are.
The fact is, the most effective firearm to keep for self defense is the one you will practice with. Having a large caliber handgun for self defense is not ideal if you are too afraid to practice with it regularly. And if you feel like you'd feel safer at home having a .44 Magnum but can barely wrap your fingers around the grip it's ok - just because you might want a smaller "gun", it doesn't have to mean a smaller caliber. The same goes for those who have large hands, you don't necessarily only have to choose large calibers. There are so many options now that will accommodate any hand size without sacrificing the ammunition size. Also, you will need to assess what your intended use will be. Will you be using it for home defense, concealed carry, or just recreation - or all three? Where do you live? Are you in an apartment surrounded by walls which would potentially have people on the other side of them? Do you live in a house alone? Do you live in the country with acreage? Are there kids in your home? How often do you plan to practice with this firearm? Can you afford to practice regularly with the caliber you chose? All of these questions are important when choosing a firearm for you.
As far as handguns are concerned, the grip is a critical aspect of selecting the right size for you. When you can get a firm and comfortable hold on the grip then the size of the caliber and the length of the barrel are both secondary. Having the proper grip is very important when practicing accuracy and consistency and also makes a huge difference when countering the recoil associated with larger calibers. For a lot of handguns you have the option of switching out the factory grip for smaller, more custom grips to fit your hands. Take this option into consideration when looking at firearms to purchase, if you like the size or caliber of the firearm but the grip is too big for your hand ask about switching them out.
You'll hear it discussed at many different ranges, pro shops, or shooting events - "knock down power" or "stopping power". And while it is true that my .45 ACP can cause exponentially more damage than your .380, your .380 can cause infinitely more damage than using nothing at all if it is what you are comfortable shooting and practicing with. A large caliber handgun in the hands of somebody who is uncomfortable with it is a lot less effective than a much smaller caliber in the hands of a skilled and confident firearm owner.
One of my most preferred handguns for home defense is the .357 Revolver simply because of the effectiveness and versatility of the caliber, as well as the simplicity-of-use of the Revolver. The .357 Revolver comes in many different makes, models, calibers, and sizes. You can find one with a a short, stubby barrel, a long barrel, or anything in between based on your preferences and needs. It is nice to have the option of shooting .38 Special ammo at the range for practice, which has less recoil and is more affordable, and then keep .357 Hollow Points in it at home for protection(the .357 can shoot .38 Special OR .357 Magnum). IF you ever do have to use it to defend yourself, grabbing it and shooting it is quick and simple. You don't have to think about whether the magazine is in it or loaded, is there a round chambered, whether the safety is on or off, what to do if it jams or misfires... you simply pull the trigger and if it doesn't go "bang", then pull the trigger again and it will. I recommend a Revolver for people who want a handgun but struggle with having the hand strength to manipulate the slide on a semi-auto, or for the person that can't practice enough to be completely familiar with the semi-automatic handgun in any scenario.
The caliber for you is the caliber you need based on what you intend to use the firearm for. If plinking away at the range is all you want to do, something mid-sized like a 9mm might be a good match. If the cost of ammunition is a concern for you but you want to practice often, then something versatile like the .357 Magnum mentioned above, or small and economic like a .22 might be a good fit. If all that matters is stopping somebody who is intruding and intending harm in your home, then a larger caliber is preferred as long as you can practice enough to be comfortable and proficient with it. If you walk into any Firearm shop and take a look at the variety of ammunition you will notice that there are many different calibers, most of which can be comparable in size or price if you're looking at the mid-size rounds. Like I've mentioned, the most effective caliber is the one you are most comfortable practicing with.