Yes, weapons are dropable. Instead of Ctrl + D in No Remorse, the key combination is changed to Ctrl + L in No Regret. No Regret allows you to have all the guns available at the same time in combat. Reason for you can not pick up a gun is that you have already got another gun of the same type in your possession.
I think the drop feature is intended for those joystick users. It's painful thing to cycle through 12 guns in combat, while most of them are useless. You can drop the weapon useless for you to make life easier using joystick. However, unlike No Remorse, in No Regret, if you drop a gun in base, continue your quest, the dropped gun will be unrecoverable after the following mission.
I always get rid of SG-01A as soon as I get AC-88 in Mission 4.